Build Your Own Language, Part 5

If you’ve followed this series up to this point, you now have a lexing and parsing engine that, while not doing much, is passing its tests. What we don’t have is a way to really say that our language works because there’s no runtime. That’s what I’ll start to address in this post.

If you think of most languages you know that are class-based — C#, Ruby, Java, Python — then you will probably recognize that they tend to have some root level object that all other objects, including classes, derive from. In these languages, pretty much everything — or literally everything — is an object. Even classes are really just a type of object. In Ruby everything really is an object. In Java that’s not quite true as there are so-called primitive types that are not objects.

Continuing with the test-first focus that I’ve been using since the start of this series, let’s create a test spec for the runtime. Create a file called runtime_spec.rb in your spec directory. Start it off with this:

And … here’s where I sat baffled initially. What do I test for?

Well, I know I want to test that my Movie class can be created. But at the runtime level, what I’m really testing for is that a Class can be represented in memory. So in looking at Ruby, you’ll often hear that “everything is an object” and that’s true. Even classes are an object. To see this:

irb(main):001:0> puts Class.ancestors
Class
Module
Object
Kernel
BasicObject
=> nil

The Module and Kernel are modules and not part of the direct inheritance, so you can ignore those. What you can see is that a Class is a type of Object. If you want to prove this to yourself, do the following in IRB:

irb(main):002:0> Class.is_a? Class
=> true
irb(main):003:0> Class.is_a? Object
=> true
irb(main):004:0> Object.is_a? Class
=> true

All classes are an instance of Class. That includes the class Class! It seems there’s a circular relation there. Anything that’s a class — including a Class! — is an instance of Class. But, in fact, they are all part of the super Object. So I figurd that’s where I’d start: with some high-level object that will be the basis for all objects. (This is similar to how java.lang.Object serves as the base for all objects in Java.)

Since everything is an object in my language, everything I will put in the runtime needs to be an object. The top level will be ToyLangObject. So everything in my runtime will ultimately be an instance of this class. Again, keep in mind: every object in the ToyLang will have a class. So for my test I want to assert that the runtime class of an object of type Class is a Class. I’ll update the test accordingly:

So I’m saying that the runtime_class property of an object of type Class (as determined by the runtime) should be a object of type Class (again, as determined by the runtime).

I suppose I should try and make that test pass at this point, huh? In your project directory, create a runtime directory. In the runtime directory, create a file called object.rb. In the object.rb file, start with the following:

The ToyLangObject is the root of our language’s structure. Everything is ultimately a ToyLangObject. Even classes are a type of this object. Or, put another way, classes are a class of object. Since ToyLongObjects have a class and since an instantiation of the class will allow me to test for what type of object it is, let’s consider how our Movie object is going to be instantiated:

class Movie
{
}

film = Movie.new

So I need to make sure that ToyLangObject can be instantiated. That’s my ultimate goal but for right now I just have to make sure that the notion of the class Movie can be represented as a class. That will pave the way for allowing Movie to be instantiated. Add the following to the object.rb:

So here I’m putting in that runtime_class property that I will need. The trick is that none of this will exist in the runtime unless it’s loaded or “bootstrapped” to use a common parlance. So in your runtime directory, create a loader.rb file and put the following in place:

The first statement allows me to have ToyLanguage recognize a Class. The second statement will (ultimately) allow me to check that the class of Class is Class. Put in code-speak, that would be:

Class.class = Class

That’s exactly what Ruby allows. The third statement allows an object to be of type Class. That means I could do the following:

myObject = Class.new

The fourth statement allows me to make sure that the class my new object is a type of class. In other words, something like this:

myObject.class = Class

Again, this is all stuff that Ruby can do. So I’m really just making sure that my language can do the same thing.

But what about that Runtime line? I put that in place because that’s what my test is defining as the interface. That’s great — but what actually goes there? This stopped me up for a bit. So I went back to the idea of what runtimes do.

Runtimes evaluate a language to determine how to provide the behavior. You’ll sometimes here about an “evaluation environment” or an “evaluation frame.” What it really means is the “area” where a given block of code is evaluated to determine how it should be handled and what it should do. One of the things such an environment or frame will do is determine the current class that any code is operating on. So I need a mechanism to provide this evaluation context.

In your runtime directory, create a eval_context.rb file and put the following in place:

Here I’m just making sure that when the evaluation context is checked, the current class can be determined based on the runtime_class of “self” — which should be what made the call the evaluation context in the first place.

Okay, now back to the loader.rb file. We can fill in the Runtime:

So what that does is creates the Runtime object. This is the root context of all things in terms of the ToyLanguage. All code evaluation will start with this root. So now I can call out specifics. Add the following to loader.rb:

Here I’m specifying what “Class” should return, which is the reference to toylang_class, which is in turn referring to a new instance of ToyLangClass.

In order to see if the test has any hope of passing, I have to add in some require statements to get my runtime elements available. So in runtime_spec.rb, add the following:

If you try to run the test right now, you’ll be told that you are not passing the correct number of arguments. This is true. In the loader.rb file, the first thing executed is a statement that attempts to instantiate ToyLangClass. So I need an initialize method in the ToyLangClass. That much is clear. What’s not clear is what has to go in there. Let’s try this in object.rb:

This gets me further in that if I run the test now, I’m told that I’m missing a new method on the ToyLangClass. Given that this is true, I’ll add that:

That creates a new instance of this particular class.

I run the test again. Arrgh! More failure. Now I’m told that there’s an undefined []= method on my EvalContext class. This is because of the following line in loader.rb:

Runtime["Class"] = toylang_class

Okay, so let’s stop a minute here and think about what we’re seeing and doing. The loader.rb file is being used to bootstrap the runtime. All the classes and objects that need to be present in order for the runtime to work are established here. So what happens is that we have this line being executed:

Runtime = EvalContext.new(object_class.new)

The object_class is a reference to a new instance of ToyLangClass, so really the statement is saying this:

Runtime = EvalContext.new(ToyLangClass.new)

In eval_context.rb, the EvalContext method is initialized as such:

EvalContext(current_self, current_class)

So given how it’s being called, the initialization is really this:

EvalContext(ToyLangClass, ToyLangClass.runtime_class)

So there’s our operating context. The runtime also establishes this about a class:

Runtime["Class"] = ToyLangClass.new

It’s that call requiring me to have a method that allows an array to be assigned to on the EvalContext class. So I’ll define that in eval_context.rb as follows:

I’m not sure what’s going to go there yet. What I do know is that since I need a way to set elements in the array, I’ll need a method to read from the array as well. In fact, my test in runtime_spec.rb is already counting on such a method being in place. So I’ll add one more thing to eval_context.rb:

Again, I’m not sure what should go there yet. Okay, so let’s run that test again. Please tell me this works!

Nope. What you should see now is this error:

  1) Testing the Runtime should determine that a class is a class
     Failure/Error: Runtime["Class"].runtime_class.should == Runtime["Class"]
     NoMethodError:
       undefined method 'runtime_class' for nil:NilClass

So what’s happening here is that the runtime_class is being called on a non-existent (nil) class. That means Runtime[“Class”] does not contain a reference to anything.

This part was not obvious to me. I felt that this suggested that the ToyLangClass is not actually returning anything useful when it is instantiated. However, it really had to do with those [] and []= methods I just defined. Let’s consider how I want to reference things:

Runtime["Class"] = ToyLangClass.new
Runtime["Class"].runtime_class

In the former case, what the []= method will be setting is something like this:

Class, #<ToyLangClass:0x2d96e58>

The [] method will simply getting this:

Class

These are basically values that I need to be globally accessible to my runtime. That means that I should make sure to set any such values passed in to the [] or []= methods to a class constant. So here are the changes that have to be made to EvalContext:

That — finally! — should get the test passing.

I know this has been a long slog through a lot of material. But we need to make sure we solidify our understanding of why this works. So let’s add another test:

That second test will fail. The failure will be:

  1) Testing the Runtime should determine that an object is a class
     Failure/Error: Runtime["Object"].runtime_class.should == Runtime["Class"]
     NoMethodError:
       undefined method 'runtime_class' for nil:NilClas

Now think about what we did so far. Try to figure out why this is happening. The issue here is that when something called Object is referenced in the Runtime, it has no runtime_class. Why would it not?

Because … we have to define that Object concept in the loader.rb, which populates the runtime. So add this line to loader.rb:

Now your test should pass.

Let’s set up another test that we know should pass; namely that the objects we are referencing cannot be nil. Add the following test:

Finally, let’s make sure that we can create a new instance of the sole elements that make up our runtime. Add the following test:

Those tests should pass.

So this is great, right? We have a lexer and a parser. And now we have this runtime thing. How cool is that? Well, it would be great — except we still really can’t see anything running. The full picture is not there and our toy language is thus nothing more than theory. What we need now is an interpreter. That will be the subject of the next post.

Share

About Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.
This entry was posted in Language Building. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *