I’ve talked a bit about Serenity in previous posts as well as Cucumber-JVM. Here I’ll combine the two and talk about how to set up a simple Cucumber-style project. Along with this focus, I’ll be concentrating a bit more on the Serenity reporting as part of this. I should note that I’m one of those people who started off being a cautious advocate of Cucumber, even writing a clone of the tool that I called Lucid, including an entire series of blog posts on it. I’ve since come to believe that Cucumber can create a lot more problems than it actually solves. My personal feelings aside, however, many testers still find themselves in the Cucumber camp, either by choice or simply because they are in an environment that has made the choice for them. I do think the “pull English down” concept of a tool like Cucumber has an interesting corollary with the “push English up” approach of Serenity. So even if you don’t like Cucumber, considering those two aspects by way of example is usually a good idea. As I mentioned in other posts on Serenity, the main thing I’m hopefully providing here is context and nuance around work that already exists. I have cribbed a lot of material from various tutorials, YouTube videos, the official Serenity User’s Guide, as well as existing GitHub repos that hold Serenity demonstration projects. My goal is not to provide anything “new” here, per se, but rather to reframe what exists in a way that I hope will make it palatable for readers of varying experience and skill level. As with my Serenity Screenplay posts, I’ll be focusing on the TodoMVC application for purposes of my demonstration code.

Setting Up the Build

In previous posts I’ve focused on Gradle as the build solution. Here I’ll use Maven. This can be helpful when starting out because a Cucumber archetype is available to help you with the setup of a new project. I’m going to provide you with a specific pom.xml file for this post so you don’t need to generate the archetype. But in case you are curious, you can use the archetype by filtering on Serenity, like this:

mvn archetype:generate -Dfilter=serenity

You should see some option like this:

net.serenity-bdd:serenity-cucumber-archetype

It will likely have some description like “Serenity automated acceptance testing project using Selenium 2, JUnit and Cucumber-JVM.” Again, though, here I’m just going to show you my own pom.xml file, shown in its entirety here: Ah, the joys of bloated XML, huh? Admittedly there is some extra stuff in there that isn’t strictly necessary for this post, but I’ve made my POM file similar to what you’ll see in the official Serenity Demo projects, so there is less cognitive friction when you are looking at official material.

The Plugins

Depending on your level of Maven knowledge, all of what I say here may be well-known to you. If that’s the case, feel free to skip forward. Serenity integrates well with Maven due to the use of a Maven plugin for Serenity, which you can see is included in the above POM file, at lines 138 to 158. As part of the settings for the plugin, note that I generate the Serenity aggregate reports during the post-integration test phase and specify the aggregate goal to generate those reports. The reason this is being done is because, in certain contexts, you’ll want the Serenity tests to run as integration tests, which means during the integration-test phase of the Maven build. This would be as opposed to running the tests as unit tests. This is why I’m using the maven-failsafe plugin, which handles integration tests. Of particular note here is that I don’t want my build as a whole to fail at the first test that fails. Rather, I want all tests to be executed. This is what allows Serenity to actually build up an aggregate report. If any tests have failed, a build failure will be triggered during the verify phase of the lifecycle. Again, this is all provided by the maven-failsafe plugin, which Serenity is being used with. In the default configuration of the maven-failsafe plugin, only files matching a certain pattern will be considered integration tests. These are files that follow the patterns **/IT*.java, **/*IT.java, and **/*ITCase.java. For acceptance testing, particularly when using tools like Cucumber, you won’t want to be bound to that. Here I’ve configured the maven-failsafe-plugin to run all of the files that end with the extension .java. An admittedly brute force approach but, in this case, it gets the job done. Others would argue that you would apply more specific patterns like **/*Spec.java or **/*Story.java and so on. To run tests and generate the reports, you would just do this:

mvn clean verify

This will run the tests and generate an aggregate report in the target/site/serenity directory. As a brief side note on this, I have set the maven-surefire-plugin (on lines 103 to 110) to skip tests although that’s not strictly necessary. The surefire plugin kicks in when you do an mvn clean test rather than the above command. The reason I have this plugin skip tests is because even though I’m going to be calling the “verify” goal, the “test” goal will be called since it’s part of the overall chain that leads up to a “verify” goal. The maven-surefire plugin is what is executed during the “test” goal. This plugin looks for a different set of default files than maven-failsafe. Specifically, surefire is looking for files with these patterns: **/Test*.java, **/*Test.java, and **/*TestCase.java. Here I’m making sure that if there do happen to be any such files, they will be skipped for execution, at least by Surefire. Finally, you do need to add the Serenity dependencies to your project. You will typically add the Serenity core and then some other dependency that corresponds to the testing library you are using. I happen to be using Cucumber here so I have to make sure to add the Serenity Cucumber plugin to the project. Lines 73 to 77 show that as part of the dependencies. You might wonder why I’m including the Serenity JUnit dependency as well on lines 68 through 72. Technically, I don’t need to for what I’m doing here. I tend to include this because it can give me a little flexibility with certain tests that I might want to run that won’t go through Cucumber. In other words, I might have high-level scenarios that are executed via Cucumber but then I have lower-level checks that I might want run as part of the test execution, but not tied to specific feature files.

Cucumber

If you’re interested in this post, you probably already know about Cucumber. But just to be somewhat thorough, let’s talk about it a little bit. Cucumber-JVM is the Java-based version of Cucumber. Serenity provides integration with the tool. With Cucumber you write scenarios in feature files. These story files contain what are generally called the “acceptance criteria”, although there’s a term that begs for and receives a lot of debate. The format of the story files is based on a structural API known as Gherkin. Much of these files are written in the plain English that you want to express your scenarios in. The structural elements — such as keywords like Scenario, Given, When, Then — are used to allow the files to be instrumented such that they can be passed into automation tools for execution. There is plenty of information out there about Cucumber in general so I won’t focus too much on it as a tool here. Rather, let’s instead focus on how you put Cucumber to use within the context of Serenity.

Features

These feature files can be placed in different locations, but you can reduce the amount of configuration if you place them in a location that Serenity will expect them to be in. In a Maven (and Gradle) build structure, you’ll have a src/test/resources directory. Within that directory, create another called features. You’ll typically organize the feature files in subdirectories that reflect your higher-level requirements. So within the features directory, create another called record_todos. By default, Serenity supports a simple directory-based convention for organizing your requirements. The standard structure uses three levels: capabilities, features and stories. A story is represented by a Cucumber .feature file so two directory levels underneath the features directory is considered “enough” as part of the conventions of Serenity. Within the directory you just created, create a file called add_new_todos.feature. Put the following in it:

Feature: Add new todos
  Users need to be able to quickly add tasks as fast as they can think of them.

  Scenario: Add a new todo
    Given the todo application
    When  the todo action 'Digitize Supreme Power Collection' is added
    Then  'Digitize Supreme Power Collection' should appear in the todo list

If you have worked with Gherkin-style feature files before, you’ll note that I have a scenario but above that I have a small section that could be considered a narrative for this particular feature file. This information will never be turned into an executable context, as the scenario will, but it can serve as a bit of context for the overall feature. Of note for my purposes here is that this bit of narrative will be included in Serenity reports. Now let’s run our build and see what kind of reports we get. Execute this:

mvn clean verify

Once this runs and then open the file at target/site/serenity/index.html. Let’s take a moment to check out what the report is telling us.

Aggregation Report

This report is broken into tabs. The first tab is called “Overall Test Results” and it provides information about test statistics. Right now we don’t actually have any tests so this tab will devoid of useful information. There is also the “Requirements” tab. When we have tests as part of our code base, all tests results will be organized as associated with requirements. You will also see a sub-table called “Capabilities”. You’ll see that the only item on this table is called “Record todos” and I bet you can guess where that came from: it’s from our directory called “record_todos”. The “Capabilities” tab simply provides that sub-table as a full table in its own right. There is also a “Features” tab. This page lists all the features that are part of your suite. The only entry on this table will be “Add new todos” and, as you can no doubt guess here as well, that’s come from the name of the file called “add_new_todos.feature”. If you expand that row you’ll see the bit of narrative text that is part of the current feature file. What you won’t see are the sceanrio along with the Given/When/Then steps. The main reason for that is because the scenario part is the test and that test is not being executed yet and thus not being reported on. You might wonder what the point of this is. If you read some of my previous posts about Serenity, one of its core ideas is to promote the idea of living documentation. So what I’ve shown you so far is that if you store your feature files in a convention-based directory structure, Serenity can use the information in that structure to reflect requirements and features. This is the case even if no tests have been created for those features. The idea being that you can have a form of living documentation that is automatically capable of being updated when new information is provided.

Hierarchy

With the aggregate report I’ve shown you, the hierarchy breakdown, provided by convention, is that everything derives ultimately from requirements. Within your requirements you will have capabilities and then features that provide those capabilities. You might not like the idea of that breakdown. Let’s say you want to break things down by epics instead of capabilities. Create a file in the root of your project called serenity.properties. Then put the following in it: Now run the mvn clean verify command again and you’ll notice that now, in the aggregate report, the “Capabilities” tab is replaced by an “Epics” tab. But what if you want to have another layer breakdown, called “ability”? Change the line in serenity.properties to this: Now let’s add another directory layer to represent the “ability.” Within the src/test/java/resources directory, we already have features/record_todos. Let’s add a directory between those two, so that the structure is: features/manage_todos/record_todos. Now run the build again and check the report. Now you’ll see that you have a tab for Epics (with entry “Manage todos”), Abilities (with entry “Record todos”), and Features (with entry “Add new todos”). I don’t want to belabor this point too much here except to show that it is possible to use some conventions to break down your reporting based on common notions of how requirements are described and broken down. Just to be clear for going forward, here’s what my current src/test/resources structure looks like:

Narrative and User Stories

Another aspect to these directories and the living documentation is the idea of a narrative. If you have a special narrative file in any of the above mentioned directories, Serenity can pull that information in as part of the overall documentation. To test this out, create a file called narrative.txt and place it in the “record_todos” directory. You can put whatever you want in that file. Here’s an example:

Recording todos

Users must be able to record tasks quickly and easily.
Users must be able to see that tasks are active, which means not completed.
Users should also be able to complete tasks, which would make the tasks inactive.

Notice here that we’re also working towards building up a bit of a ubiquitous language. We’re talking about domain concepts like “completed” and “active.” The idea of “not completed” seems synonymous with “active” while “completed” would seem to be synonymous with “inactive.” That could lead us to other questions, such as whether there is any other reason a task may be inactive. Or whether a completed task could be “reactivated” by indicating it is no longer complete. That’s the point of user stories: to have discussions about the feature without necessarily going overboard with documentation. These stories should have just enough detail to spur on further discussion about various bits of functionality, including edge cases. Refining the domain language is also possible. Perhaps, for whatever reason, the Product Team does not want anything referred to as “inactive”, and instead prefers “In Progress” or “Complete”. You could then make sure that your stories reflect that preferred wording. The living documentation can then always be produced to be reflective of these decisions. To see this, run the build again (mvn clean verify) and check out the report. Assuming you followed my examples above in terms of adding “epic,ability”, you will see on the “Abilities” tab that the entry “Recording todos” can now be expanded and it will show the narrative text. The same would apply for any of the high level concepts if you put a narrative.txt file in the appropriate directory.

Summary

One thing I want to make clear is that what I’ve shown you here about the directories and the aggregate report is not specific to Serenity’s integration with Cucumber-JVM. Much of what I said here applies if you are integrating with JBehave as well. It also can apply if you are simply using JUnit as your runner. The aggregate reporting features of Serenity are meant to apply separate and distinct from the particular runner you are using. What we have done, however, is start to utilize this concept within the context of Cucumber-JVM. We did this by providing a feature file and we saw at least the beginning of this file being utilized in the reporting. In the next post, we’ll get into the actual execution. I realize this post may seem a little disappointing in that, title notwithstanding, I didn’t do much with Cucumber here. Again, however, I wanted to set up the context. Further, I wanted to make it clear that certain key elements of Serenity, like the reporting, have little to do with tools like Cucumber itself. I closing I want to note that there is an interesting interplay of concepts here, which manifests as a “push or pull” distinction. In this case, the “push” part would be pushing from code to English. The “pull” part would be pulling down from English to code. In the case of tools like Cucumber, you are definitely pulling. As I showed in my screenplay posts, however, you can entirely utilize a push strategy. Yet it’s not an either-or. You can do both.