Previously I had talked about the craft of testing, focusing on the balance between the creativity of an artist and the methodology of a scientist. In both cases, however, the focus is on communication. To that end, it’s imperative we can express what we test in good English. (Or whatever the native language of your environment is.) So let’s talk about this key skill.
The ability to write your tests in both descriptive and prescriptive natural language is a key skill of testers. This skill is a necessity regardless of how much and to what extent you rely on test automation. The thing I believe so-called xSpec and xBehave solutions got right is the focus on natural language expression to guide testing.
Even though it’s important to write tests in good natural human language, this writing can be constrained because writing tests is all about breaking down into conditions and cases. While it’s not useful to speak of “positive testing” and “negative testing”, it is very useful to speak of valid and invalid conditions. This is what your natural human mind is going to think in terms of and thus what your natural human language is most often going to express. You are going to take test conditions and apply data conditions to them. Some will be valid, some will be invalid. Further, all testing ultimately breaks down to a simple “input –> action –> output” formula, which is easily handled by any human language on the planet.
Refactoring Tests
I’m going to break down an existing test I was presented with and convert that into one of the popular structured test formats, relying on Given/When/Then. Such formats have a set of conditions that are covered by a scenario. So here is the original summary that was provided on the test case:
Original summary: Verify costs/fee dependent on CRF pages
This was for a clinical trial business domain. I changed that slightly to read this:
Scenario: Verify Effect of CRF Pages on Fees and Cost
Is this a case of “you say to-may-to, I say to-mah-to”? Possibly. I felt my change was a little more accurate and grammatically correct, which does matter.
The Context (Given)
The first steps in the original test case are somewhat interesting:
1. Create a plan with any phased study 2. Go to Treatment tab 3. Keep Trial Design as 'Parallel' 4. Select 'Yes' for 'Is this an End Point Study?'
Those steps are all talking about a context. The context is a clinical plan. So the immediate question I asked the team is: “What’s our domain phrase for this?” Lots of discussion had already taken place around this and I ended up going with one possible idea that everyone seemed to agree on:
Scenario: Verify effect of CRF Pages on Fees and Cost
Given a plan with a parallel trial design for a late stage endpoint study
Now, please keep in mind: you may not be able to see how I got from the steps to the statement. You don’t have the business domain knowledge that you would have were you working in that environment. But do notice that what I did was simply remove the imperative action steps and made a statement about what was being created when those steps were followed. This would allow the UI to evolve without having to change the test at all. All that said, it’s important to keep in mind that none of those steps were the test. Those steps just provided the context for the test.
Here was the next step in the test:
5. Fill in all details, Note the value for field 'Average CRF pages collected per extended visit:'
Here’s an example where — at least apparently, based on how the tester wrote the test — it doesn’t matter what “details” we use, as long as we use some. So my question to the team was: “Do we want to specify data here? Or allow it to be vague like this?” There are pros and cons to each approach, as you can imagine, and I discussed those with the team.
Also I asked the team why the “note the field” instruction was in there. The field, as it turned out, defaulted to blank. So what was I “noting”, exactly? As it further turned out, you have to put some value in. Ah, so, I asked the team, “Does this mean ‘note the value that you put in when you fill in all details'”?
The answer was ‘yes’ but now take notice that all of this is still part of the context. While “fill in the details” is an action, it’s not the action we are testing for.
In any event, from a test expression standpoint, now we have a branch point. Do I write with explicit values or not? Well, to help the team along, I went with both routes for demonstration purposes. Let’s consider what explicit values would mean.
Scenario: Verify effect of CRF Pages on Fees and Cost
Given a plan with a parallel trial design for a late stage endpoint study
And with a treatment arm with the following
| Weeks from FSI to endpoint | 52 |
| Subjects will visit every | 2 |
| Treatment duration | 6 |
| Number of CRF pages per subject | 100 |
| Visits per subject | 5 |
| Average CRF pages per extended visit | 20 |
Those are obviously quite specific values. If we don’t use specific values then, presumably any values will do. I’ll explore that later.
Reader Exercise: What are the benefits of using the specific values? What are some problems with using specific values? As context, I’ll tell you that there were many other values necessary to create a plan. But these particular values were the only ones that I called out. So what does that imply for our test data and how we express it?
Supporting Data (Given?)
Here was the next step from the test:
6. Note Summary tab values: Fees and Cost
Okay, I duly noted these summary tab values. The problem was that there were two costs listed so I could have presumed that what was meant was Total Fees and Total Study Costs, since that’s what the fields were called. The point, I said to the team, is that the test should remove ambiguity. Here’s what I saw on the screen of the application:
• Total Fees = 114,026,851 • Total Study Costs = 147,079,035
I don’t know how or even whether to specify that yet, so I left it alone for the moment. This is definitely still part of the context though. It just happens to be values that are calculated as part of the context. So step 6 was really just an instruction to the tester to keep something in mind.
The Action (When)
Here was the next step.
7. In Treatment tab, Increase value for the field 'Average CRF pages collected per extended visit:'
Okay, so now we finally get to an action. This is the reason we are doing this test. I want to see the effect of this action. Notice here that the mandate is just to “increase.” We could say that. We could also say what to increase by. So I suggested that the team could do this:
When the Average CRF pages collected per extended visit is increased
Or they could do this:
When the Average CRF pages collected per extended visit is increased to 40
Well, since I’m already in specific mode, let’s continue the trend:
Scenario: Verify effect of CRF Pages on Fees and Cost
Given a plan with a parallel trial design for a late stage endpoint study
And with a treatment arm with the following
| Weeks from FSI to endpoint | 52 |
| Subjects will visit every | 2 |
| Treatment duration | 6 |
| Number of CRF pages per subject | 100 |
| Visits per subject | 5 |
| Average CRF pages per extended visit | 20 |
When the Average CRF pages collected per extended visit is increased to 40
The Observable (Then)
Here are the next steps from the test:
8. Verify Summary tab values: Fees and Cost 9. Fees and Cost should decrease from the values noted in step 6
Now I have an observable. As I pointed out to the team, since the original test states everything relatively, you can’t know what the values will be at this point. You can simply know that they should decrease. But decrease from what? Well, what we were told to “note” earlier.
I told the team that we should give a shot at stating all our data conditions, so, by that logic, we should state the fees and costs as well.
Scenario: Verify effect of CRF Pages on Fees and Cost
Given a plan with a parallel trial design for a late stage endpoint study
And with a treatment arm with the following
| Weeks from FSI to endpoint | 52 |
| Subjects will visit every | 2 |
| Treatment duration | 6 |
| Number of CRF pages per subject | 100 |
| Visits per subject | 5 |
| Average CRF pages per extended visit | 20 |
And with the following summary values
| Total Fees | 114,026,851 |
| Total Study Costs | 147,079,035 |
When the Average CRF pages collected per extended visit is increased to 40
Reader Exercise: What’s the potential gotcha in what I specify for the fees/costs versus the level of detail I specified for the rest of the plan?
As per the action I was stating, I tried it out in the application. I made the change to 40 and here is what I ended up with:
• Total Fees = 175,754,694 • Total Study Costs = 210,853,158
I’ve taken you through a lot here, but … did you catch it?
I noticed the opposite of what the original test was telling me. The fees and costs have increased, not decreased. So the original test appears to be wrong. Specifically, what I saw and showed to the team:
• Total Fees: New value = 175,754,694 Old value = 114,026,851 • Total Study Costs: New value = 210,853,158 Old value = 147,079,035
Anyway, I could now add my observable:
Scenario: Verify effect of CRF Pages on Fees and Cost
Given a plan with a parallel trial design for a late stage endpoint study
And with a treatment arm with the following
| Weeks from FSI to endpoint | 52 |
| Subjects will visit every | 2 |
| Treatment duration | 6 |
| Number of CRF pages per subject | 100 |
| Visits per subject | 5 |
| Average CRF pages per extended visit | 20 |
And with the following summary values
| Total Fees | 114,026,851 |
| Total Study Costs | 147,079,035 |
When the Average CRF pages collected per extended visit is increased to 40
Then the summary values will increase to the following:
| Total Fees | 175,754,694 |
| Total Study Costs | 210,853,158 |
That’s a nice specific test. Notice that the ‘Then’ (observable) doesn’t just list the values but says they will increase. Again, as I pointed out to the team, I believed the original test is stated incorrectly so I modified it.
Now the final steps of the test:
9. In Treatment tab, decrease value for the field from step 5, Verify Summary tab values: Fees and Cost 10. Fees and Cost should increase from the values noted in step 6
So we have another action and observable. We handled that like this:
Scenario: Verify effect of CRF Pages on Fees and Cost
Given a plan with a parallel trial design for a late stage endpoint study
And with a treatment arm with the following
| Weeks from FSI to endpoint | 52 |
| Subjects will visit every | 2 |
| Treatment duration | 6 |
| Number of CRF pages per subject | 100 |
| Visits per subject | 5 |
| Average CRF pages per extended visit | 20 |
And with the following summary values
| Total Fees | 114,026,851 |
| Total Study Costs | 147,079,035 |
When the Average CRF pages collected per extended visit is increased to 40
Then the summary values will increase to the following:
| Total Fees | 175,754,694 |
| Total Study Costs | 210,853,158 |
When the Average CRF pages collected per extended visit is decreased to 10
Then the summary values will decrease to the following:
| Total Fees | 83,327,132 |
| Total Study Costs | 114,580,244 |
Again, I corrected here for what appeared to be a mistake in the original test.
So that’s a key point here. By reframing the test, with an example we found a discrepancy that was in no way obvious by looking at the test itself. But notice that I only noticed this when I actually tried it out in the application. The practice of example-driven testing is predicated upon the idea that working out examples like this ahead of time can stop problems like this from cropping up.
Yet … that takes a lot of work, right? Considering that test I just walked you through, that’s a test with a lot of stated data values. Granted, it’s helpful in that it is example-driven. It gives the business rules (encoded by saying “when value is increased, other values increase”) and those rules are backed up with an example. Yet — it’s still a lot of data, right? That approach is often the most difficult and certainly the most time consuming.
Using Non-Stated Values
If I do all this without stated values, I end up largely with this:
Given a plan with a parallel trial design for a late stage endpoint study
When the Average CRF pages collected per extended visit is increased
Then the summary values for Total Fees will increase
And the Total Study Costs will increase
When the Average CRF pages collected per extended visit is decreased
Then the Total Fees will decrease
And the Total Study Costs will decrease
That definitely just gives me the business rule without any specific data. But notice this minimum counts on the tester to make some assumptions. Clearly if a computed value is going to increase or decrease, you should note the value before making the change. The test above does not state that. That, of course, is more in line with a requirement. A requirement is not going to admonish you to make sure you check something first. It’s going to tell you what should happen in a given context.
Another thing to notice is that there is an absence of information as to where anything is. If someone doesn’t know that the Average CRF pages collected per extended visit is on the Treatment tab of a plan, they will have to do some searching. Likewise, someone has to know where to check for the Total Fees and Total Study Costs.
You could modify the test accordingly:
Given a plan with a parallel trial design for a late stage endpoint study
When the Average CRF pages collected per extended visit on the Treatment tab is increased
Then the values for Total Fees on the Summary Tab will increase
And the Total Study Costs on the Summary tab will increase
When the Average CRF pages collected per extended visit on the Treatment tab is decreased
Then the values for Total Fees on the Summary Tab will decrease
And the Total Study Costs on the Summary tab will decrease
Here you run into lots of duplication of wording. You could try to table it out, I suppose:
Given a plan with a parallel trial design for a late stage endpoint study
When the Average CRF pages collected per extended visit is increased
Then the following values on the Summary tab will increase
| Total Fees |
| Total Study Costs |
When the Average CRF pages collected per extended visit is decreased
Then the following values on the Summary tab will decrease
| Total Fees |
| Total Study Costs |
You can probably see other variations.
And Thus Is There Art…
What I’m trying to show here is something I covered only very peripherally in how testing is an art and a science. Being able to wordsmith, if you will, is an important skill of a tester.
Notice how there are many little micro-decisions that I made in the example above as I engaged with the team. I covered another such example regarding collaboration and test writing.
I’m sure many of you didn’t fail to notice that I’m using a Gherkin-like approach here but, as I talked about in “Why Cucumber? Why Gherkin?”, there is value to the above structured approach to test writing, regardless of the extent to which you are having those tests be directly automated.
The point is the levels of abstraction that you will move between as you consider business rules and business understanding, which means considering how to state intent, and reveal implementation. That leads into further considerations of how data driven to be and whether to opt for example-based statements of data or more generalized data conditions. This forces you to start considering heuristics. For example, don’t encode what you don’t know in a test. You can encode the fact that you don’t know in a test so it’s manifestly quite clear that you have something that isn’t testable. Another heuristics is that when you have specific data values, it should be clear why you are using a specific data value rather than any value or any value within a range.
Finally, assuming you read this far, you might be thinking: “Wow, that post was a lot of work to trudge through.” It was mainly because I wanted to provide a real example. And real examples have lots of details and can get messy. But this level of work is what is necessary to be an effective and efficient tester, particularly if you believe, as I do, that testing is a design activity (even at the business level) and that the primary value of tests are their ability to act as a communication mechanism.
That is the craft of testing.
