I have been working on a different type of tool solution that I call Specify. In this post I’ll introduce the tool.

Regarding my motivations for this tool, I have lately been falling into the camp of those who are detractors of tools like Cucumber and SpecFlow. (And this is even after writing my own clone of such tools, called Lucid.) Yet while I’m in process of abandoning the Cucumber club, I do still maintain that there is a benefit to having natural language requirements-as-tests that are intent-revealing and implementation-proving. Specify is my attempt to keep the parts I like and abandon the parts I don’t.

First, let’s consider a simple breakdown of the differences between my Cucumber-clone, Lucid, and Specify.

Lucid (alternative to Cucumber, SpecFlow, Lettuce, Behat)
Input: structured natural language specifications
Action: parsing and execution of an AST
Output: delegation to matchers, which encapsulate code which delegates to tools

Specify (extension of RSpec)
Input: structured code-based specifications
Action: execution of blocks
Output: delegation to code

Some Setup

Let’s get the setup tasks out of the way first.

• Intall Specify: gem install specify
• Install Symbiont: gem install symbiont
• Create a spec directory.
• Within the spec directory create a support/models directory.
• Within the spec directory create a file called spec_helper.rb.
• Within the spec directory create a file called planet_weight_spec.rb.

Specify leverages RSpec as its test execution runner. Cucumber (and Lucid) provided their own executables that you had to call. Specify does not. You simply execute the RSpec tool as you normally would and if your repository requires the Specify gem, then execution will leverage the Specify extensions and additions.

In the spec_helper.rb file, add the following:

What this does is require some elements that you are going to need for this post. Specify is of course the main thing I’m showing here so requiring that is fairly obvious. Symbiont is a driver library I wrote for automation. The factory being included is a context factory mechanism that you’ll see in action. Do note that you do not have to use Symbiont with Specify. You can use any driver library you prefer. The final line in the code above simply makes sure to require any files found in the support directory.

With that, we can get started.

Coding a Spec in Specify

Using Specify, here is an example of what you can put in the planet_weight_spec.rb file.

Keep in mind that we’re using the RSpec defaults: a spec directory, a spec_helper.rb file, and a test file named with the pattern *_spec.rb. This means you would run this just as you would any other RSpec test. Doing so would get you the following output:

$ rspec

Planetary Weights
  Calculate Weights for Rocky Planets
    Mercury
      Given the weight page (PENDING)
      When a user enters a weight of 200 (PENDING)
      Then their weight on Mercury will be exactly 75.6 (PENDING)
      And  their weight on Mercury will be approximately 70 (PENDING)

Finished in 0.00052 seconds (files took 0.36731 seconds to load)
1 example, 0 failures

What Specify does is put a Gherkin-like wrapper around RSpec but without necessarily adhering to Gherkin’s limitations. Specify does not in fact use Gherkin as a dependency. As one example of what this freedom affords, in Gherkin Feature and Ability are two top-level keywords. You can’t use them together. As you can see with Specify, you can use the Ability to further refine an aspect of the Feature.

In RSpec terms, Feature and Ability are example groups and would correspond to describe or context in standard RSpec usage. (I do have a brief wiki article describing RSpec operation if it helps.)

The Scenario is similar to an example step that can itself contain steps. The Scenario, Given, When and Then keywords are directly equivalent to an RSpec it block.

You can see that I use a tag, just as RSpec would, at the scenario level. The tag here is :phantomjs, which is meant to indicate something about how the example should execute. In this case, the tag indicates that the example should be run using the PhantomJS browser. This, of course, only matters if your driver library knows how to use PhantomJS.

Upon execution, these steps show up as pending right now because you did not include a block. If you included the block, then all of the steps would pass since there is nothing to tell them what failure looks like. If you want to include the blocks but keep the steps pending, you can use an RSpec tag just as you would with regular RSpec usage:

Comparing Specify to Cucumber

Comparing all this for a moment to tools like Cucumber or my own Lucid tool, here’s an example of a feature file that would contain the same information:

Feature: Planetary Weights

  @phantomjs
  Scenario: Mercury
    Given the weight page
    When  a user enters a weight of "200"
    Then  their weight on Mercury will be exactly "75.6"
    And   their weight on Mercury will be approximately "70"

You would then provide step definitions to handle those phrases:

So with tools like Cucumber you need two files for the specification. With Specify you need one. However, do note that the steps in the Cucumber world are re-usable and parameterized. The Specify steps are not parameterized nor are they, at least in the current context, very reusable short of simply using them again in another file. Do note, however, that this would not give you ambiguous match errors as Cucumber would.

There are ways to mimic some of what you do in the Cucumber world in terms of sharing steps but I’ll hold off on discussing that for now.

Execute Tests with Specify

Now let’s consider how the Specify example can be filled out with details. Just as with the step definitions in a Cucumber-like approach, your example statements in Specify would delegate down to some driver to carry out specific actions.

In my case, I’m going to use my own Symbiont tool to showcase this. Symbiont is predicated upon the idea that you are using a domain model pattern. In the case of web applications, the domain model would be a page object. For web services, the model would be an activity object. In the support/models directory, create two files — domain.rb and weight_page.rb. Put the following in each file:

domain.rb weight_page.rb

All of this has to do with Symbiont so I won’t cover it too much here. I have posts on how Symbiont works and there is a Symbiont Wiki as well. Do note that if you want to play along, the above page is part of my Dialogic app. You can either view Dialogic on Heroku or download a local copy. Instructions on the Dialogic home page tell you what to do. If you decide to use the remote site, please note you’ll need to change the localhost address in the url_is declaration above.

So now let’s fill out the Specify spec file we started with in order to utilize this page model. I’ll do a couple of iterations of this. Here’s a start:

Here I’m using Symbiont to call up the page, perform some actions on the page, and then check some values. Notice the use of the @page instance variable that you are using between each step. That’s important.

If you know RSpec, you know that sharing information between example steps is not possible. So another thing that Specify is providing is the ability to treat RSpec tests as integration tests rather than just unit tests. In the above case, the @page instance variable exists for the duration of the “Mercury” scenario.

If you were to run this, you would get an error like this:

Failures:

  1) Planetary Weights Calculate Weights for Rocky Planets Mercury
     Failure/Error: on_view(WeightPage)
     NoMethodError:
       undefined method `goto' for nil:NilClass

The stack trace would show you that the error is coming from Symbiont and the problem is that it’s trying to run tests against a browser but you have not specified a browser. To be clear: you have tagged the example as using the PhantomJS browser, but that’s just a tag. Something must actually act upon the tag. So let’s add the following to your spec_helper.rb file:

Here I use the RSpec configure block to use before and after hooks to indicate that examples with a particular tag (:phantomjs) should perform some steps before and after execution. Specifically, in this case, the before action tells Symbiont to instantiate a PhantomJS browser. Do note that you have to have the browser installed; Symbiont doesn’t provide it for you; it just wraps it if it exists.

Assuming you have PhantomJS on your system, you should now see the tests passing with all the above in place. If you want to verify that this is actually working, feel free to change one of the expectations. For example, if you change the Then step to expect a different weight value, you’ll get this output:

Planetary Weights
  Calculate Weights for Rocky Planets
    Mercury
      Given the weight page
      When a user enters a weight of 200
      Then their weight on Mercury will be exactly 75.6 (FAILED)
    Mercury (FAILED - 1)

Failures:

  1) Planetary Weights Calculate Weights for Rocky Planets Mercury
     Failure/Error: expect(weight).to eq(75)
       
       expected: 100
            got: 75.6
       
       (compared using ==)

That should give you confidence that the tests are actually calling the Planetary Weight page and the JavaScript is being executed to generate the results.

Incidentally, if you wanted to see an actual browser instead of running headless, you could replace the :phantomjs tag with :watir. Then add the following to your spec_helper.rb file:

One thing I want to make clear here is that I have tagged the Scenario with :phantomjs. The Scenario in Specify is equivalent to it in RSpec. This means that the block being run is of type example. That’s why the hooks above specify :example.

Thinking About Expectations

This next bit is not so much relevant to Specify, since it would apply to Cucumber and Lucid, but it is relevant to making sure tests have a good and effective level of expressivity combined with using consistent test design patterns.

You might note that in the Then and And steps I’m directly referencing a particular field on the page. To be clear, consider this statement:

This is referring to “mercury” which corresponds to the following element declaration on the WeightPage class:

So the statement in the Then steps is getting the value of that field and then performing expectations regarding the value contained by that field. This is essentially asking the page for a bit of information about itself. It’s effectively tying you into a specific detail: the exact identifier used in the page object that represents the ‘mercury’ text field. So another way to do this is to simply wrap the expectation around a specific action provided by the page. Here’s a revised example, showing just the observable steps:

This does the same thing as the previous logic but calls a method defined on the page instead.

Something to note is that all of these examples so far are relying on the example step to provide the expectations and/or assertions after asking the page for something. Let’s try one more variation:

This is more of a “tell, don’t ask” approach. Here the page is being told to do something: confirm weight for passed in parameters. This means the expectations are now part of the domain model object. Those expectations will only work, however, based on the data conditions passed to them from the test.

Neither approach is right or wrong. Specify will not force one or the other upon you. Simply choose what you feel makes the most sense.

Adding to the Spec

Now let’s add another planet to the mix.

This reuses all of the logic from the page model so everything should work as-is. At this point, however, the spec is getting a little busy. A lot of information is being repeated. In the Cucumber world you would use a background to DRY up the spec. You can do exactly that with Specify but with a slight twist. Here’s an example:

You’ll notice there are two backgrounds. One happens before all examples are run (which fires up the browser and goes to the page) and another happens before each individual example, which performs the common action of calculating the weight of a 200 pound person.

However something interesting happens here. In Cucumber it would still work to tag the scenario because the background gets imported into the scenario behind the scenes. That’s not quite what happens in Specify. So if you ran the above example it would fail because the Background(:all) has run first and no browser is established until the scenario is executed and the :phantomjs tag is encountered.

One thing you could do is put the tag at the Ability level:

This would allow you to remove the tag from the scenario level for each scenario in your test. That removes a small bit of duplication, which is nice.

However, doing this means you do need to add some configuration options to spec_helper.rb. Here is an example, including a little cleanup of this section of the file:

What this does is allow the phantomjs tag to be handled for context blocks (which Ability is) as well as the example blocks it was already handling. I’m also doing this for the watir tags as well.

With this abstraction of the Given and When steps into backgrounds, I purposely left something out. Notice that our output might read a little weird because all of the context happens in the background. For example, running the spec as I showed you above results in output like this:

$ rspec spec/gui/planet_weight_spec.rb

Planetary Weights
  Calculate Weights for Rocky Planets
    Mercury
      Then their weight on Mercury will be exactly 75.6
      Then their weight on Mercury will be approximately 70
    Venus
      Then their weight on Venus will be exactly 181.4
      Then their weight on Mercury will be approximately 180

Finished in 5.6 seconds (files took 0.34624 seconds to load)
2 examples, 0 failures

Unlike Cucumber, with Specify, not every step must have a reportable aspect. You can see that when I moved the code from the Given and When steps, I simply copied the executable code and not the actual Given and When statements. Yet I could retain the original output simply by not removing the step descriptors as I did. For example, the backgrounds could be changed as such:

Okay, so now let’s add another ability to the spec:

Now, once again, we find ourselves with duplication. Specifically we are duplicating the :phantomjs tag again and my backgrounds are entirely duplicated in each ability block. In this case, we can move our backgrounds to the Feature level (outside of the ability). Do note that this will require moving the :phantomjs tag to the feature level. So here’s what your spec could refactor to:

You should find that all of this works.

Wording Your Spec Examples

Now I want to go back to the wording. Let’s change the before blocks to where they just execute the code rather than output context information:

Now as I mentioned earlier, doing this makes the Then statements read a little odd. There is no context necessarily. Abstracting out as much output as possible is something that allows me to consider how better to frame my test spec. After all, most of that information is simply saying “a 200 pound person will weigh x on some planet.” That’s really all my test results need to convey.

Further, saying “Then” doesn’t really add much in the case of these scenarios because there’s no Given or When. When you want to simply specify something and have as clean of output as possible you can use a specify keyword:

This will give you the following output:

Planetary Weights
  Calculate Weights for Rocky Planets
    Mercury
      a 200 pound person will weigh 75.6 pounds on Mercury
      a 200 pound person will weigh approximately 70 pounds on Mercury
    Venus
      a 200 pound person will weigh 181.4 pounds on Venus
      a 200 pound person will weigh approximately 180 pounds on Venus
  Calculate Weights for Gas Giants
    Jupiter
      a 200 pound person will weigh 472.8 pounds on Jupiter
      a 200 pound person will weigh approximately 470 pounds on Jupiter

Using the specify suppresses any keyword output and simply gives you exactly what you specified.

Even with this, however, there’s still a lot of detail in the spec and it’s possible for things to get out of sync. The English statements, after all, don’t directly map to the code as they would in Cucumber or Lucid. Let’s consider one technique to get around that.

Iterate Over Conditions

You can create structures that you can iterate over. Here’s an example of how you could entirely modify the spec we have been working on:

One thing you sacrifice there a bit is the readability of the step itself. However, the data conditions themselves are certainly visible and are factored to reveal the bare minimum necessary:

Planetary Weights
  a 200 pound person will weigh 75.6 on Mercury.
  a 200 pound person will weigh 181.4 on Venus.
  a 200 pound person will weigh 75.4 on Mars.
  a 200 pound person will weigh 472.8 on Jupiter.

A nice aspect to this is that when you want to add planets, you would simply add them to the lists. The steps would continue to work for each entry.

Output Without Execution

One benefit of Cucumber is often stated to be that the test spec is input to the code. That means you always have a readable representation. Clearly with a tool like Specify, the spec is part of the code. Further, the more code there is, the harder it can be to discern the test spec itself. That’s particularly the case with the last example we just looked at. The code itself could be hard to read although the output itself is fine.

So what if I want to get that output but without having to execute? You can do that with a simple addition to the rspec command:

rspec spec/planet_weight_spec.rb --dry-run > spec.txt

That will get you just the spec contents — stored in the file spec.txt — but without actual execution.

Welcome to Specify

This was moderately detailed, whirlwind tour of how to use Specify. There is a project wiki for Specify that I will continue to update and eventually put some structure to. Specify leverages RSpec and that means you can leverage the RSpec ecosystem as well, such as various extensions or output formatters that others have made available.

Future posts will discuss some of how my focus shift from a Lucid-approach to a Specify-approach informs how I write tests and provide those tests to others.