The goal of a Test Description Language (TDL) is to put a little structure and a little rigor around effective test writing, where “effective test writing” means tests that communicate intent (which correspond to your scenario title) and describe behavior (the steps of your scenario). Since those attributes should be what all statements of requirement strive for, this means that requirements and tests, at some level of approximation, can be the same artifact. That “level of approximation” is the point at which you get down to specifying the behavior that users find value in.
Let’s dig into this a bit more.As you build up your sets of scenarios, you will initially encode intent and you will gradually reveal implementation. Some like to argue that there is a conceptual shift away from “individual test cases” to a “test specification.” I would rather think of these as feature specifications and remove the notion of them being “test cases”, “test suites” or “test specifications.” I say that because I think it’s important to get testers to think in terms of functional suites of behavior that are based on feature, and elaborated by scenario.
Let’s Consider an Example
Let’s say the Valve Corporation is using its Steam game delivery service to have one of their weekend sales. The Valve stakeholders want to implement a feature that will give discounts to customers based on what type of customer they are.
The business analyst provides this:
If the Customer Type is Active and Total Purchase is less than or equal to $10.00 Then do not give a discount Otherwise, give a 1% discount If Customer Type is Gamer Then give a discount of 1% for any order If Total Purchase is greater than $50.00 Then give a discount of 5%
Is this good enough for a feature specification? Well, let’s ask it this way: is this good enough for testing? There is at least one bit of missing information in there where you could make an assumption.
You might take a moment here and think about what that is.
Refine the Example
Let’s reword the above business rules a bit more in Given-When-Then:
GIVEN an Active customer WHEN the customer makes a purchase less than $10.00 THEN no discount is applied WHEN the customer makes a purchase of exactly $10.00 THEN no discount is applied WHEN the customer makes a purchase of $10.50 THEN a discount of 1% is applied
Okay, wait. I have a When…Then, When…Then, When…Then. Do I have a test smell here? Should I perhaps break this up?
Refine the Example (Again)
Scenario: No Discount For Active Customer GIVEN an Active customer WHEN the customer makes a purchase less than $10.00 THEN no discount is applied WHEN the customer makes a purchase of exactly $10.00 THEN no discount is applied Scenario: Discount for Active Customer GIVEN an Active customer WHEN the customer makes a purchase of $10.50 THEN a discount of 1% is applied
Does that work? Let’s continue then.
Expand the Example
Scenario: Discount for Gamer Customer GIVEN a Gamer customer WHEN the customer makes a purchase of any value THEN a discount of 1% is applied Scenario: Discount for ?? GIVEN a ?? customer WHEN the customer makes a purchase greater than $50.00 THEN a discount of 5% is applied
Ah. There’s our possible assumption, right? Does that last business rule apply to a customer who has Active as well as Gamer?
When giving this exercise to testers, I find many tend to blissfully breeze right by the assumption and either assume that the last rule applies to Gamer or assume it applies to both Active and Gamer. Either interpretation could be correct. But if you don’t know, you don’t know. Don’t encode what you don’t know in a test. Encode the fact that you don’t know in a test so it’s quite clear that you have something that isn’t testable.But … wait? Isn’t there something we’re missing? Some other detail that might matter?
Think about it for a bit before moving on.
Question (and Refine) the Examples
One question I would have is: are the second and third rules additive? What if a Gamer (automatic 1% discount for ANY order) makes a purchase of $60.00 (giving them their 5% discount)? Do they then have a 6% discount applied? Again, many testers breeze right by that.
Let’s say these questions are taken back to Valve (the customer, remember) and decisions are made:
- Active customers will not be given a 5% discount.
- Discounts are additive.
Okay, so let’s get those business rules encoded as tests:
Scenario: Discount for Gamer Customer GIVEN a Gamer customer WHEN the customer makes a purchase of any value THEN a discount of 1% is applied WHEN the customer makes a purchase greater than $50.00 THEN a discount of 5% is applied
Well … yeah. But it’s not really clear that the rules are being applied, right? How about this:
Scenario: Automatic Discount for Gamer Customers GIVEN a Gamer customer WHEN the customer makes a purchase of any value THEN a discount of 1% is applied Scenario: Gamer Customers Get a High Discount for Big Purchases GIVEN a Gamer customer WHEN the customer makes a purchase greater than $50.00 THEN a discount of 5% is applied
Okay … but what about that additive part? Do I have a different scenario? Would I have this:
Scenario: Gamer Discounts are Additive GIVEN a Gamer customer WHEN the customer makes a purchase greater than $50.00 THEN an automatic discount of 1% is applied AND a purchase discount of 5% is applied
Stop and Consider the Examples
First, note that here we have introduced some domain terms: “automatic discount” and “purchase discount”. It’s easy for those to just slip in during collaboration. It’s even easier when someone is writing these things on their own. On the other hand, do I need to call out that it’s additive? Someone might say why not just make a scenario that says a 6% discount is applied?
There are very good reasons for breaking them out, I believe. But I’ve seen many testers do exactly what I just said: have one scenario that doesn’t talk about ‘additive’ anything and simply says that a 6% discount will be applied to Gamer customer orders over $50.00.
Fine … but if I have the “additive” scenario do I really need the “high discount for big purchases” scenario?
Well, see that’s the interesting thing isn’t it? I’m technically making clear two things here: a larger discount is applied to Gamer customers for big purchases. Yet I don’t really have anything that contrasts this. I don’t have a scenario that shows that an Active customer spending over $50.00 simply gets the 1% discount. I have the scenario that says when an Active customer makes a purchase of “any value.” That certainly could imply a value over $50.00 along with any other value — but should it be called out?
Wait … it could “imply a value of over $50.00”? Why? I know that the notion of a “big purchase” means over $50.00 because I read it in the business rule. Good for me — but my scenario doesn’t really call that out, does it? If you all had was my scenario to go on, how do you know that the cutoff isn’t in fact $40.00 and I just happened to choose $50.00 to make the value larger than the cutoff? What is and what is not a “big purchase” is not defined.
Consider the Expression
Before getting into data value specifics, which of those approaches above is easier to read and parse? Well, first, to be fair, let’s put the full examples together, with all the customer decisions made.
First here are the business rules stated in one structured form:
If the Customer Type is Active and Total Purchase is less than or equal to $10.00, Then do not give a discount, Otherwise, give a 1% discount. If Customer Type is Gamer, Then give a discount of 1% for any order. If the Customer Type is Gamer and the Total Purchase is greater than $50.00, Then apply a 6% discount, made up of the normal 1% discount plus a special 5% discount
Here is another structured form, this time in Given-When-Then format:
Feature: Apply a Discount for Weekend Sale Purchases Scenario: No Discount For Active Customer GIVEN an Active customer WHEN the customer makes a purchase less than $10.00 THEN no discount is applied WHEN the customer makes a purchase of exactly $10.00 THEN no discount is applied Scenario: Discount for Active Customer GIVEN an Active customer WHEN the customer makes a purchase of $10.50 THEN a discount of 1% is applied Scenario: Automatic Discount for Gamer Customers GIVEN a Gamer customer WHEN the customer makes a purchase of any value THEN a discount of 1% is applied Scenario: Gamer Customers Get a High Discount for Big Purchases GIVEN a Gamer customer WHEN the customer makes a purchase greater than $50.00 THEN a discount of 5% is applied Scenario: Gamer Discounts are Additive GIVEN a Gamer customer WHEN the customer makes a purchase greater than $50.00 THEN an automatic discount of 1% is applied AND a purchase discount of 5% is applied
Take a moment to look at each collection. What seems to make sense?
The Art of Expression
I brought up some questions above regarding what I should and should not call out. And therein lies the art of constructing tests — whether of the type of expression you are using. Arguably you are always using a test description language of some sort. The question is whether and to what extent your TDL is structured and what consistent principles guide the expression of tests.
My questions so far are predicated upon boundaries or making data values clear. But when you have specific data values, it should be clear why you are using a specific data value rather than any value or any value within a range.
Consider the Data Conditions
For example, consider the scenario of the automatic discount for a Gamer customer. Some would argue that you should start at the smallest possible total to test this out: $0.01. Consider the scenario where I make sure that an Active customer making a purchase “less than or equal to $10.00” results in no discount? Should I be more specific? Should I test more at the boundaries?
Some would suggest I should test at $9.99 (no discount applied) and at $10.01 (discount applied). Likewise, consider the scenario of the purchase discount for a Gamer customer when their purchase is “greater than $50.00”. Should I be more specific? Some would say you should show this happens just after the $50.00 point: $50.01.
Notice here that you are starting to consider a breakdown of the scenarios based on specific data conditions. This is what I think drives the desire to have a more structured TDL, such as the Given-When-Then format. Granted, business analysts can write up more business rules but they are essentially doing so as a specific type of narrative. Is that what you want? Well, again, take some time to actually consider the differences in the above formats.
It’s this act of considering alternatives that I find most testers fall down on. They simply want to get something done rather than thinking hard about the type of expression.
Incidentally, some would argue that you can make the above even simpler. Consider:
| Customer Type | Purchase Total | Discount Applied | | Active | $9.99 | 0% | | Active | $10.00 | 0% | | Active | $10.01 | 1% | | Gamer | $0.01 | 1% | | Gamer | $10.00 | 1% | | Gamer | $50.00 | 1% | | Gamer | $50.01 | 6% |
Here we’re sort of settling around boundaries although, again, that’s not clear. We still lose some of why the values are in place, right? Is it clear that the reason for data rows 4 through 6 is that the Gamer customer gets a 1% discount for any value purchase? Is it clear from data row 7 that the 6% is made up of automatic discount (for being a Gamer customer) plus the ‘big purchase’ discount?
Could I rectify this? How about with this:
| Customer Type | Purchase Total | Discount Applied | | Active | less than or equal $10.00 | 0% | | Active | greater than $10.00 | 1% | | Gamer | greater than $0.01 | 1% | | Gamer | less than $50.00 | 1% | | Gamer | greater than $50.00 | 6% |
This might help call out why specific number ranges are being used, but there’s still some bits missing here, right? I still see the fact that I have a 6% discount, without any knowledge that this was because a 5% purchase discount was added to the 1% automatic discount.
Don’t Lose the Business Rule
These tables are certainly more concise but we might risk losing the business rule or at least aspects of it. Even the tactic of some test writers to include the table as part of the scenario may only make sense in certain cases. For example, with the BDD scenarios I listed, consider this one:
Scenario: No Discount For Active Customer GIVEN an Active customer WHEN the customer makes a purchase less than $10.00 THEN no discount is applied WHEN the customer makes a purchase of exactly $10.00 THEN no discount is applied
That one could potentially become:
Scenario: No Discount For Active Customer
GIVEN an Active customer
WHEN the customer makes a purchase of
| less than $10.00 |
| exactly $10.00 |
THEN no discount is applied
But, then again, I could also just say:
Scenario: No Discount For Active Customer GIVEN an Active customer WHEN the customer makes a purchase of less than or equal to $10.00 THEN no discount is applied
And, in fact, that’s pretty close to what we said here in the original business rule:
If the Customer Type is Active and Total Purchase is less than or equal to $10.00, Then do not give a discount,
Expressing the Business Rules
So then it really comes down to saying that if we are going to make such a short shift from the business rule as written to the test scenario, then why not just write it that way to begin with? That is what you do with a TDL: you communicate intent and you describe behavior. You essentially turn a business rule into business understanding. Some would argue that you do that by taking acceptance criteria and turning them into acceptance tests.
What about this business rule?
If Customer Type is Gamer, Then give a discount of 1% for any order.
That became:
Scenario: Automatic Discount for Gamer Customers GIVEN a Gamer customer WHEN the customer makes a purchase of any value THEN a discount of 1% is applied
Again, you could argue, we may as well write it in the more structured form to begin with. A question in both cases is whether we consider “any value” effective enough for a test? Why not spell it out like in the table format? Well … why? I would actually like a scenario like this that has a bit of fuzz testing built in. After all, each tester may try a different value. And we might learn something from that. For example, what if some tester makes a purchase that is less than $0.00?
Refine Business Understand by Exploring Business Rules
Wait … what’s that now? Less than $0.00? Why would there be an item on the Steam library that is less than $0.00. There probably wouldn’t be but an astute tester is really implicitly asking: what should happen if Purchase Total is less than $0.00?
Lo and behold, the business users say that this could in fact happen because a type of rebate coupon or gift card might be used that is greater than the total of the items that the customer is purchasing. So in these cases the discount applied should be 0% if the purchase total is less than $0.00. And this should apply to any customer. This seems worth calling out in a scenario.
Scenario: No Discount Applied For No Dollar Amount Purchases GIVEN an Active customer and a Gamer customer WHEN the customer makes a purchase of any value AND applies a rebate coupon greater than the purchase price THEN no discount is applied WHEN the customer makes a purchase of any value AND applies a gift card greater than the purchase price THEN no discount is applied
Feels kind of busy, right? At least it does to me. The “No Dollar Amount Purchases” phrase in the scenario title is being used to convey any mechanism that could lead to such a condition happening. Further, this general condition means I have a multi-faceted context: an Active customer and a Gamer customer. If we add more ways for a “No Dollar Amount Purchase” or if we add more customer types this can get cumbersome.
Further, we want to change this condition only for one customer and not another, now I have to break them out. Beyond that, while these are related observables conceptually, there are different tests being done here. Even further, what if we have rebate coupons or gift cards that do not cover the entire purchase price?
Scenario: Gamer Customers Get No Discount with Rebate Coupons That Lead to No Dollar Purchase GIVEN a Gamer customer WHEN the customer makes a purchase of any value AND applies a rebate coupon greater than the purchase price THEN no discount is applied Scenario: Active Customers Get No Discount with Rebate Coupons That Lead to No Dollar Purchase GIVEN an Active customer WHEN the customer makes a purchase of any value AND applies a rebate coupon greater than the purchase price THEN no discount is applied Scenario: Gamer Customers Get No Discount with Gift Cards That Lead to No Dollar Purchase GIVEN a Gamer customer WHEN the customer makes a purchase of any value AND applies a gift card greater than the purchase price THEN no discount is applied Scenario: Active Customers Get No Discount with Gift Cards That Lead to No Dollar Purchase GIVEN an Active customer WHEN the customer makes a purchase of any value AND applies a gift card greater than the purchase price THEN no discount is applied
Is that any better?
Scenario Titles are a Guide
One thing that may be nice is that at a glance our scenario titles can be read individually to see the permutations we are considering.
Scenario: Gamer Customers Get No Discount with Rebate Coupons That Lead to No Dollar Purchase Scenario: Active Customers Get No Discount with Rebate Coupons That Lead to No Dollar Purchase Scenario: Gamer Customers Get No Discount with Gift Cards That Lead to No Dollar Purchase Scenario: Active Customers Get No Discount with Gift Cards That Lead to No Dollar Purchase
This is handy for business, developers, and testers to make sure that the appropriate data conditions are being considered. In fact, let’s consider all of our scenarios just by title alone:
Feature: Apply a Discount for Weekend Sale Purchases Scenario: No Discount For Active Customer Scenario: Discount for Active Customer Scenario: Automatic Discount for Gamer Customers Scenario: Gamer Customers Get a High Discount for Big Purchases Scenario: Gamer Discounts are Additive Scenario: Gamer Customers Get No Discount with Rebate Coupons That Lead to No Dollar Purchase Scenario: Active Customers Get No Discount with Rebate Coupons That Lead to No Dollar Purchase Scenario: Gamer Customers Get No Discount with Gift Cards That Lead to No Dollar Purchase Scenario: Active Customers Get No Discount with Gift Cards That Lead to No Dollar Purchase
The scenario title tells you, at a glance, what is different about this scenario from every other scenario. Could I tighten this up this language even further to make it easier to tell at a glance the intent of each scenario? Probably. It’s worth an exercise, I suppose.
The key point I want to show here is that short, concise intentional behavior is what should be described. If your scenario titles become massively long, you are missing the spirit of intent and are probably including implementation.
Remember that “big purchase” concern I talked about? Here is what I had:
Scenario: Gamer Customers Get a High Discount for Big Purchases GIVEN a Gamer customer WHEN the customer makes a purchase greater than $50.00 THEN a discount of 5% is applied
Some might argue that the scenario title should have been this:
Scenario: Gamer Customers Get a High Discount for Purchases Over $50.00 ...
What do you think? The good thing is it makes the intent clear, right?
Avoid Incidentals
Well, actually, no — or it does make intent clear, but with incidentals. What we need to understand is that we call a “big purchase” is a concept that is a specific condition that triggers behavior. It just so happens that our specific value for this is over $50.00. And that is called out in the steps. However, I argue that it should not be called out in the scenario title because the intent (“big purchase”) is different from the specific data condition for that intent (“greater than $50.00”).What this might argue for is changing the wording. We might feel that “big purchase”, in the context of this feature, is too vague. So the title becomes:
Scenario: Gamer Customers Get a High Discount for Qualifying Purchases ...
Is that any better? That could actually lead to problems if you have different types of qualifying purchases, such as, say, New Games, Premium Content Games, Game Collections — whatever else.
Evolve the Expression Language
But this highlights a point: you will evolve your language as you go. I started out with one descriptor (“big purchases”), I eventually evolved that to another (“qualifying purchases”) and I’m already thinking why that might not be good enough either. But I only make those changes — those spec refinements — when I have information that leads me to make a responsible choice.
Speaking of refinement, one thing to understand here is that these scenario titles — and the debate that goes around constructing them — could be written during a spec workshop. This could be during the discovery phase or whatever else you want to call it or it could just be whenever people get together to discuss. The actual steps that describe the behavior — and how the intent is satisfied — could be written out during a further spec workshop. This could be during an elaboration phase.
Regardless of when the steps are constructed to flesh out the scenario, the steps should only be speaking to the intent as described in the scenario title. If your scenario title is kept short and concise, that tends to be a lot easier to do. Those final four scenarios in my above list are getting about as large as I would want to consider because scenario titles are all about conveying the intent for a defined action and observable.
Again, as I said earlier, I have encoded intent with these short, concise scenario titles. I will gradually reveal implementation as those scenarios are filled out with steps that written in terms of what the user is doing or experiencing in order to get value.
Keep The Specification Coherent
The final important point here is that all of this is being done in the context of a specific feature specification. That feature file will have a feature title at the top of it. Every scenario you write should be a way of stating one bit of intentional behavior of that feature; it is a way that the feature can be used. If your feature is too broadly stated, you’re going to have a lot of scenarios and potentially a lot of confusion. If your feature is too narrowly stated, you may end up with only one scenario! Once you have those scenario titles, then your behavioral steps — written in the language of the domain tasks of your users — must all showcase a key action and an observable to that key action.
What you probably noticed here is that I’m not just giving you rules; I’m giving you heuristics. Constructing an expressive TDL that is also executable does not require a rule book. It requires an engaged team thinking about not just how to communicate now, but how to make sure that the communication stays relevant and useful as various aspects of an application evolve over time.
That said, I’m finding “better” and “worse” things you can do when you construct your tests as part of a TDL and I’ve covered at least my opinions on some of those areas above. This is an area where my thinking is still very much in an evolutionary phase. My purpose in this post was simply to give everyone some insight into my thinking on this subject.
