If you read some tutorials online, what you’ll find out about RSpec is that you’ll have constructs like this:

Here the describe() method is used to define an “example group.” The string I pass to the method represents the facet of the system that I want to describe. In this case, a search made on the Amazon site.

The it() method defines an example. The string passed to the method describes the specific behavior I’m interested in specifying about that facet. In this case, I’m interested in seeing that the search results displays a range of results.

My focus is on testing web sites and web applications. As such, I’m not using RSpec as a tool to help me test Ruby objects. I’m going to use RSpec to wrap the execution of Watir. So this means that any of my Ruby logic will go in the it() methods. That logic will then express the intent described in the combination of the describe() and it() methods. What do I mean by the intent? Well, let’s start on a full example. Start off with this in the testSpike_03.rb file:

Now run this command:

rspec testSpike_03.rb --format documentation

The output will be:

An Amazon search
  should display results range
  should reflect the search context in the title

The documentation format outputs the name of each behavior (“Amazon search”) and each example of that behavior that I want to check. That’s the intent of this test. If it helps, think of the describe statement as being a test case and each example as being a test condition on that case. My Watir commands now have to be put in place to execute a test that reflects the intent. I already wrote the tests in testSpike_01.rb so I just have to transplant them to my new spike file. These tests, incidentally, are referred to as specifications in RSpec. Here’s what I ended up with:

To run the script, do this:

rspec testSpike_03.rb

There really isn’t an easy way (that I’ve found anyway) to pass arguments to an RSpec script. So I dropped the “specify the browser at the command line approach” for now. So to try this on different browsers, just modify the statement that specifies the browser to use. Note you have to do this in both it() methods. And that brings up a good problem: the fact that I have browser handling calls (highlighted in the above example).

Before getting into that, here’s the output you should see from running this script:

$ rspec testSpike_03.rb
..

Finished in 43.55 seconds
2 examples, 0 failures

Wait, what was that I was saying about getting better output? This looks like quite a step back. I’m still looking at the console to determine what happened and I’m given even less information. The two dots indicate success of two tests. If one of the tests failed, you would see an F in place of the dot. Still, this leaves quite a bit to be desired. Before I even continue with RSpec, I want to know if I can get better than this. It turns out I can. Try running the script with this command:

rspec --format html --out results.html testSpike_03.rb

That will give you a nice HTML page. Notice that the HTML result page is only indicating the intent level statements. A green color indicates that the test passed. Any tests that failed would be showing up in red. I can still see lots of room for improvement here but this gives me hope. So let’s look at some of the logic in the script so that I can make my test logic a bit better.

I would prefer if the logic to instantiate the browser and navigate to the Amazon web site were only stated once. I’d like those to be executed before each test begins. Similarly, the browser should close after each test but, again, I’d only like to state that one time. I could just make separate methods and have each test call those methods. However, RSpec providse a better solution. Modify testSpike_03.rb so that it looks like this:

Here I’ve added before and after methods. The before(:each) statement is being used to provide a starting state for each example (i.e., each it method). The after(:each) statement is saying what the end state or cleanup should be. Again, since I’m saying :each this means what will happen after each example (it method) executes. The practical effect of this with the above script is the same as the one I had before, except that I’m repeating myself less. Specifically, before each example is executed, a browser will be opened up and the Amazon site navigated to. After each example is executed, the browser will be closed.

I could have used before(:all) and that would mean the statements in the before method would be run only once for the entire example group. The same applies if I were to use the after(:all) variation. In general, it’s usually considered a more effective practice to use before(:each) because that re-creates the context before each example and keeps state from one example getting into another example. Put another way, it makes sure your tests stay independent with each test relying on a well-defined context.

This approach does, as you’ll notice, force me to reuse the same logic to execute the search in Amazon in each example. That’s yet more duplication. I’d like to correct that. One approach along these lines is to use helper methods that you can define in the example group. Those methods are then accessible from all the examples in that group. Here’s how that looks:

As with all such refactorings, this should have no effect on the output of the test script. Now let’s go back to focusing on the output. The output, as you’ve seen, is going to give you a dot (console) or a green title (HTML) if the test passed. The output will give you an F (console) or a red title (HTML) if the test passed. In my previous Watir script that didn’t use RSpec (that is, testSpike_01.rb), it was probably pretty clear how my verification logic was working. Let’s consider that in isolation:

Now compare the code I’m using to do this in the RSpec version:

Those “should” statements are the test verifications and they are an important part of how RSpec works. Specifically, the should statements are referred to as Expectations and work with what are called Matchers. The should expectations are not really reflected in the output. This is really just a variation of the problem I was trying to solve: I can tell that certain things ran and the pretty green color would seem to indicate they ran okay. But what actually was executed? Do I have to read the test logic to figure that out?

This gets into learning how you can format the test results to be a little different. First of all, you might want to check what failure looks like if you haven’t already. To do that, just change some of the text string content in one (or both) of the should lines from testSpike_03.rb. Then rerun the script. Here’s an example of what a failure might look like in the console:

$ rspec testSpike_03.rb
.F

Failures:

  1) An Amazon search should reflect the search context in the title
     Failure/Error: @browser.title.should == "Amazon.com"
       expected: "Amazon.com"
            got: "Amazon.com: star wars: Books" (using ==)
     # ./testSpike_03.rb:24:in 'block (2 levels) in <top (required)>'

Finished in 47.35 seconds
2 examples, 1 failure

Failed examples:

rspec ./testSpike_03.rb:21 # An Amazon search should reflect the search context in the title

In the HTML version, you basically see that same information but it just looks a little prettier although, I would argue, potentially a lot more “busy.” What I ideally want from my test results is that anyone can read them. What that means is someone should be able to glean the overall functionality being tested, the intent of a specific test, and the specific information about why the test failed.

RSpec uses message formatters to generate all of the output you see when running your specifications. These formatters receive notifications of certain events. For example, when an example group (i.e., a describe block) is about to be run, a notification is sent to the formatters. When an individual example (an it block) fails or passes, a notification is sent to the formatters. RSpec has a couple of built-in formatters. One of those is designed to generate plain-text output while another is designed as an HTML formatter. We just used the latter, in fact.

If none of the built-in formatters satisfies your specific reporting needs, you can create a custom formatter. It seems that the easiest way to do this is to subclass an existing formatter. In my case, I’ll do this with the HTML formatter that’s built in to RSpec. Create a file called testFormatter.rb. Put the following in the file:

There you go! You have a custom formatter. Invoking your new custom formatter is fairly easy. You just need to require the file in which it is defined and then add the class of your formatter to the command line. With the above file in place, for example, the command to execute the script becomes this:

rspec --require .\testFormatter.rb --format TestFormatter --out results.html testSpike_03.rb

Of course, at this point my custom formatter doesn’t really do anything. Let’s change that. First understand that there are some methods in the formatter classes that handle the output based on a certain notification. Here are some of those methods:

• example_group_started: Called as an example group (a describe block) is started.
• example_started: Called as an example (an it block) is started.
• example_passed: Called when an example passes.
• example_failed: Called when an example fails.
• extra_failure_content: Provides extra information for a failed example.

As you can see, they pretty much do exactly what they sound like. Clearly if I want to provide my own custom formatting, I probably have to figure out what those above methods actually do. Given that I’m using RSpec 2.x, I found the relevant files here:

\Ruby192\lib\ruby\gems\1.9.1\gems\rspec-core-2.6.4\lib\rspec\core\formatters

Here is what the example_passed method looks like in the html_formatter.rb file:

You can see that a lot of formatting takes place with information that is passed to the output. What I can do is now override that method in my own custom formatter. So let’s say one thing I want to do is provide more details of the verification that takes place for when an example passes. Here’s a snippet of testSpike_03.rb that shows my it methods with an addition to each:

Here I’m using metadata to define something I’m calling verify. Metadata defined on an example group is available and can be overridden from any example in that group. So that’s what I do here. What may not be obvious from the logic above is that I had to define verify as empty string metadata on the example group (at the describe level). If you don’t do this, then you will not be able to reference the metadata within each it block. (I’m not sure if that’s a bug in RSpec or how things are supposed to work. Still researching that.) Now that I have specific metadata for each example, I have to tell my custom formatter how to print that information. Here’s a modified testFormatter.rb file:

You might notice that since I include the method in my own formatter, this means the existing method (from html_formatter.rb) gets overwritten completely. So if you want all of the functionality of the original method back — like, say, the call to move_progress — then you have to make sure to put it into your overriding method. If you run this again, you should now see the verify statements show up under the intent statements. Remember the command to run with your custom formatter is:

rspec --require .\testFormatter.rb --format TestFormatter --out results.html testSpike_03.rb

I think that’s pretty slick. You’ll note as well that I did have to take care to make sure my printing of the verify metadata was done within the context of the markup that is generated for the HTML report.

What about for the failing tests? There’s a lot of information that gets presented there. Do I need all that? Well, for now let’s just say that I don’t want the “code snippet” part to show up. This is the part that is clearly some Ruby logic with some line numbers indicating where something didn’t quite work right. I rarely find this helpful and it’s usually more distracting than not. Maybe I should explain that. Let’s say the should verification in my second it block (from testSpike_03.rb) fails. Here’s what I’ll see in the HTML for that block:

    should reflect the search context in the title

    expected: "Amazon.com"
         got: "Amazon.com: star wars: Books" (using ==)

    ./testSpike_03.rb:25:in 'block (2 levels) in '

    27        end
    28
    29        raise(RSpec::Expectations::ExpectationNotMetError.new(message))
    30      end

The “code snippet” there is showing me logic that isn’t even in my test script. So why I would need to see that, I’m not sure. Certainly most of the consumers of my test results would not need to see it. That bit of content comes up due to the extra_failure_content method being called by the example_failed method. So to get rid of this, I can just create an empty extra_failure_content method in my testFormatter.rb.

This method is actually an interesting one because it can be used to add a lot of extra information to any tests that fail. So while I appear to be totally dismissing it, I will be coming back to this one a little later.

I also want to get rid of the part of the failed output that is the backtrace. In my example above this is the text that says “/testSpike_03.rb:25:in `block (2 levels) in ‘”. This usually appears right above the code snippet. To get rid of this I have to override the example_failed method. The trick here is that I only really need to get rid of one line. Below is the default example_failed method, but I have commented out the line that causes the bit of text I don’t want printed. This version of the example_failed method would have to go in testFormatter.rb.

Note that I also had to comment out the super(example) line and the reason for that is because I provide a full implementation of example_failed. So I don’t need to call the original method for anything.

Run that and see what you think. What these last examples show is that you can have a field day with changing how RSpec reports on your results. This also seems like a good place to end this post. I would recommend playing around with the solutions I’ve provided you. The main thing to note is how RSpec is being used to wrap up Watir-WebDriver execution. There’s a lot of organizing you can do within the context of RSpec describe() and it() methods. I didn’t really touch on that too much here but before getting into all that, I would recommend you feel comfortable with using the two technologies together.