Ruby Templating: Baking an Interpreter

We hope you’ve got your stroopwafels warmed on top of your coffee because today we’re gluing things up with sticky stroop (the syrup that makes the two halves of a stroopwafel stick together). In the first two parts of our series, we baked a Lexer and a Parser and now, we’re adding the Interpreter and gluing things together by pouring stroop over them.

Ingredients

Alright! Let’s get the kitchen ready for baking and put our ingredients on the table. Our interpreter needs two ingredients or pieces of information to do its job: the previously generated Abstract Syntax Tree (AST) and the data we want to embed into the template. We’ll call this data the environment.

To traverse the AST, we’ll implement the interpreter using the visitor pattern. A visitor (and therefore our interpreter) implements a generic visit method that accepts a node as a parameter, processes this node and potentially calls the visit method again with some (or all) of the node’s children, depending on what makes sense for the current node.

module Magicbars
  class Interpreter
    attr_reader :root, :environment

    def self.render(root, environment = {})
      new(root, environment).render
    end

    def initialize(root, environment = {})
      @root = root
      @environment = environment
    end

    def render
      visit(root)
    end

    def visit(node)
      # Process node
    end
  end
end

Before continuing, let’s also create a small Magicbars.render method that accepts a template and an environment and outputs the rendered template.

module Magicbars
  def self.render(template, environment = {})
    tokens = Lexer.tokenize(template)
    ast = Parser.parse(tokens)
    Interpreter.render(ast, environment)
  end
end

With this in place, we’ll be able to test the interpreter without having to construct the AST by hand.

Magicbars.render('Welcome to {{name}}', name: 'Ruby Magic')
# => nil

To no surprise, it currently doesn’t return anything. So let’s start implementing the visit method. As a quick reminder, here’s what the AST for this template looks like.

For this template, we’ll have to process four different node types: Template, Content, Expression, and Identifier. To do this, we could just put a huge case statement inside our visit method. However, this will become unreadable pretty quickly. Instead, let’s make use of Ruby’s metaprogramming capabilities to keep our code a bit more organized and readable.

module Magicbars
  class Interpreter
    # ...

    def visit(node)
      short_name = node.class.to_s.split('::').last
      send("visit_#{short_name}", node)
    end
  end
end

The method accepts a node, gets its class name, and removes any modules from it (check out our article on cleaning strings if you’re interested in different ways of doing this). Afterward, we use send to call a method that handles this specific type of node. The method name for each type is made up of the demodulized class name and the visit_ prefix. It’s a bit unusual to have capital letters in method names, but it makes the method’s intent pretty clear.

module Magicbars
  class Interpreter
    # ...

    def visit_Template(node)
      # Process template nodes
    end

    def visit_Content(node)
      # Process content nodes
    end

    def visit_Expression(node)
      # Process expression nodes
    end

    def visit_Identifier(node)
      # Process identifier nodes
    end
  end
end

Let’s start by implementing the visit_Template method. It should just process all the statements of the node and join the results.

def visit_Template(node)
  node.statements.map { |statement| visit(statement) }.join
end

Next, let’s look at the visit_Content method. As a content node just wraps a string, the method is as simple as it gets.

def visit_Content(node)
  node.content
end

Now, let’s move on to the visit_Expression method where the substitution of the placeholder with the real value happens.

def visit_Expression(node)
  key = visit(node.identifier)
  environment.fetch(key, '')
end

And finally, for the visit_Expression method to know what key to fetch from the environment, let’s implement the visit_Identifier method.

def visit_Identifier(node)
  node.value
end

With these four methods in place, we get the desired result when we try to render the template again.

Magicbars.render('Welcome to {{name}}', name: 'Ruby Magic')
# => Welcome to Ruby Magic

Interpreting Block Expressions

We wrote a lot of code to implement what a simple gsub could do. So let’s move on to a more complex example.

Welcome to {{name}}!

{{#if subscribed}}
  Thank you for subscribing to our mailing list.
{{else}}
  Please sign up for our mailing list to be notified about new articles!
{{/if}}

Your friends at {{company_name}}

As a reminder, here’s what the corresponding AST looks like.

There’s only one node type that we don’t yet handle. It’s the visit_BlockExpression node. In a way, it is similar to the visit_Expression node, but depending on the value it either continues to process the statements or the inverse_statements of the BlockExpression node.

def visit_BlockExpression(node)
  key = visit(node.identifier)

  if environment[key]
    node.statements.map { |statement| visit(statement) }.join
  else
    node.inverse_statements.map { |statement| visit(statement) }.join
  end
end

Looking at the method, we notice that the two branches are very similar, and they also look similar to the visit_Template method. They all handle the visiting of all nodes of an Array, so let’s extract a visit_Array method to clean things up a bit.

def visit_Array(nodes)
  nodes.map { |node| visit(node) }
end

With the new method in place, we can remove some code from the visit_Template and visit_BlockExpression methods.

def visit_Template(node)
  visit(node.statements).join
end

def visit_BlockExpression(node)
  key = visit(node.identifier)

  if environment[key]
    visit(node.statements).join
  else
    visit(node.inverse_statements).join
  end
end

Now that our interpreter handles all node types, let’s try and render the complex template.

Magicbars.render(template, { name: 'Ruby Magic', subscribed: true, company_name: 'AppSignal' })
# => Welcome to Ruby Magic!
#
#
#  Please sign up for our mailing list to be notified about new articles!
#
#
# Your friends at AppSignal

That almost looks right. But on a closer look, we notice that the message is prompting us to sign up for the mailing list, even though we provided subscribed: true in the environment. That doesn’t seem right…

Adding Support for Helper Methods

Looking back at the template, we notice that there’s an if in the block expression. Instead of looking up the value of subscribed in the environment, the visit_BlockExpression is looking up the value of if. As it is not present in the environment, the call returns nil, which is false.

We could stop here and declare that we’re not trying to imitate Handlebars but Mustache, and get rid of the if in the template, which will give us the desired result.

Welcome to {{name}}!

{{#subscribed}}
  Thank you for subscribing to our mailing list.
{{else}}
  Please sign up for our mailing list to be notified about new articles!
{{/subscribed}}

Your friends at {{company_name}}

But why stop when we are having fun? Let’s go the extra mile and implement helper methods. They might come in handy for other things as well.

Let’s start by adding a helper method support to simple expressions. We’ll add a reverse helper, that reverses strings passed to it. In addition, we’ll add a debug method that tells us the class name of a given value.

def helpers
  @helpers ||= {
    reverse: ->(value) { value.to_s.reverse },
    debug: ->(value) { value.class }
  }
end

We use simple lambdas to implement these helpers and store them in a hash so that we can look them up by their name.

Next, let’s modify visit_Expression to perform a helper lookup before trying a value lookup in the environment.

def visit_Expression(node)
  key = visit(node.identifier)

  if helper = helpers[key]
    arguments = visit(node.arguments).map { |k| environment[k] }

    return helper.call(*arguments)
  end

  environment[key]
end

If there is a helper matching the given identifier, the method will visit all the arguments and try to lookup values for them. Afterward, it will call the method and pass all the values as arguments.

Magicbars.render('Welcome to {{reverse name}}', name: 'Ruby Magic')
# => Welcome to cigaM ybuR

Magicbars.render('Welcome to {{debug name}}', name: 'Ruby Magic')
# => Welcome to String

With that in place, let’s finally implement an if and an unless helper. In addition to the arguments, we’ll pass two lambdas to them so that they can decide if we should continue interpreting the node’s statements or inverse_statements.

def helpers
  @helpers ||= {
    if: ->(value, block:, inverse_block:) { value ? block.call : inverse_block.call },
    unless: ->(value, block:, inverse_block:) { value ? inverse_block.call : block.call },
    # ...
  }
end

The necessary changes to visit_BlockExpression are similar to what we did with visit_Expression, only this time, we also pass the two lambdas.

def visit_BlockExpression(node)
  key = visit(node.identifier)

  if helper = helpers[key]
    arguments = visit(node.arguments).map { |k| environment[k] }

    return helper.call(
      *arguments,
      block: -> { visit(node.statements).join },
      inverse_block: -> { visit(node.inverse_statements).join }
    )
  end

  if environment[key]
    visit(node.statements).join
  else
    visit(node.inverse_statements).join
  end
end

And with this, our baking is done! We can render the complex template that started this journey into the world of lexers, parsers, and interpreters.

Magicbars.render(template, { name: 'Ruby Magic', subscribed: true, company_name: 'AppSignal' })
# => Welcome to Ruby Magic!
#
#
#  Thank you for subscribing to our mailing list.
#
#
# Your friends at AppSignal

Only Scratching the Surface

In this three-part series, we covered the basics of creating a templating language. These concepts can also be used to create interpreted programming languages (like Ruby). Admittedly, we glossed over a couple of things (like proper error handling 🙀) and only scratched the surface of the underpinnings of today’s programming languages.

We hope you enjoyed the series and if you want more of that, subsribe to the Ruby Magic list. If you are now hungry for stroopwafels, drop us a line and we might be able to fuel you with those as well!

Regular guest author Benedikt Deicke is a software engineer and CTO of Userlist.io. On the side, he’s writing a book about building SaaS applications in Ruby on Rails. You can reach out to Benedikt via Twitter.