I have been introducing Cucumber to testers who have little exposure to such tools. I was looking at whether The Cucumber Book would be worth having around the office. And while it may or may not be, one thing I notice is that it (like most resources on Cucumber I find) don’t really address some of the heuristics regarding how you can start thinking about writing test specifications.
I had talked about these concepts a little before when talking about specifying application workflow activities and writing tests as specifications. Likewise, I had talked about some general spec-writing guidelines. I want to delve more into those guidelines at some point.
Before doing that, however, I kept finding that all of this can be tricky for testers who are new to this stuff. In The Cucumber Book there is a distributed ATM example that is never really implemented but that is used as part of a multi-chapter running example. So to focus on something simple but also something specific, what I did was start with that example and break down some of the choices you have as you write test specifications. And I say that because I’m finding the cognitive hurdle a lot of testers have is the recognition of when to apply a given writing technique to a given situation based on a lack of understanding of the choices available.
So as an example, it seems clear to most people that the business rule for the ATM example is that the amount you withdraw will be subtracted from your account. So a fairly simple scenario is:
Scenario: Withdraw money from account GIVEN I have $500 in my account WHEN I withdraw a fixed amount of $100 THEN the balance of my account should be $400
That’s one way of writing it out. However, since the context (GIVEN) and the action (WHEN) can be wrapped up in a single, easy to read statement, I could say:
Scenario: Withdraw money from account WHEN I withdraw a fixed amount of $100 from an account that has $500 THEN the balance of my account should be $400
If the first-person aspects are undesirable, it’s fine to do this:
Scenario: Withdraw money from account WHEN a fixed amount of $100 is withdrawn from an account that has $500 THEN the balance of the account should be $400
It’s right here that I would tell testers to stop and think about the three variations above. They should ask themselves which they prefer. They should ask themselves which is more in line with how they think. It’s critical that testers new to this stop here and think about solutions like these. Getting used to the way you can morph the writing a bit is a critical skill to have when writing test specifications like this.
Okay, so now let’s consider how specific the test is. Do I even need to know that $500 is in the account for this? Not really, so I could say:
Scenario: Withdraw money from account WHEN a fixed amount of $100 is withdrawn from an account that has sufficient funds THEN the balance of the account should be $400
Hmm. But that doesn’t read well, does it? So in this case I may ask: do I even need the $100? Without knowing what “sufficient funds” are, a removal of $100 doesn’t tell me much and certainly doesn’t tell me that $400 is the end result. As such the $400 would also have to go to. What you end up with, essentially, is the business rule:
Scenario: Withdraw money from account WHEN a fixed amount is withdrawn from an account that has sufficient funds THEN the balance of the account should be reduced by the amount of money withdrawn
Again: here is a great place to stop. Encourage testers to think about what we just did here. My opinion is that this is a case where having the actual values may make sense. The specific example makes it clear even though the above is certainly accurate.
Assuming I put the actual numbers back in, does the example then overspecify at the price of clarity? Some testers might think that a specific example locks you in to just thinking about that example. Perhaps. For example, what the above scenario doesn’t tell you is what happens if you try to withdraw more than you have. Or when you try to withdraw when your account has no funds at all. Or when you try to withdraw with an invalid card. Let’s work out some variations.
Scenario: Attempt withdrawal using invalid card WHEN a fixed amount of $100 is withdrawn from an account that has $500 BUT the card is invalid THEN an error message appears
Now, going with our example from before, I could even combine the second clause:
Scenario: Attempt withdrawal using invalid card WHEN an invalid card attempts to withdraw $100 from an account that has $500 THEN an error message appears
Now notice here that the $500 — unlike in our other example — probably is somewhat irrelevant. Because the the balance is not changing in this case. So I could replace “an account that has $500” with a valid business domain phrase. Maybe we’re back to this:
Scenario: Attempt withdrawal using invalid card WHEN an invalid card attempts to withdraw $100 from an account that has sufficient funds THEN an error message appears
So in that case do I even need the $100? Not really because, again, unlike the previous scenario, here the actual values used do not matter as much.
Scenario: Attempt withdrawal using invalid card WHEN an invalid card attempts to withdraw money from an account that has sufficient funds THEN an error message appears
Let’s stop again for a second. Here we have two different scenarios: one that we felt needed a specific example and one that we felt served as it’s own example without the need for specific data. For many testers I work with, this flexibility of design is actually what causes problem. Many testers seem to have grown up in a context where you have some designated guideline that tells you how a test should be written. What I just showed above is that you’ll allow yourself some flexibility on this, based on the nature of what you are testing.
Let’s continue with this last example. Is the information provided in the Then clause enough? Or should I be more specific? I could be specific about the nature of the error message:
Scenario: Attempt withdrawal using invalid card WHEN an invalid card attempts to withdraw money from an account that has sufficient funds THEN an error message appears saying to contact the bank
Or I could be specific about the content of the error message:
Scenario: Attempt withdrawal using invalid card WHEN an invalid card attempts to withdraw money from an account that has sufficient funds THEN an error message appears saying """ The card you are using appears to be invalid. Your account has been made inactive. Please contact your bank, using the number on the back of your card, to resolve this issue. """
So it seems that what I’m doing is using specificity when it matters. As another example of that, let’s consider this other variation:
Scenario: Attempt withdrawal from an empty account WHEN a fixed amount of $100 is withdrawn from an account that has $0 THEN the balance of the account should remain at $0 AND an error message appears saying there are insufficient funds
Note here “an account that has $0” could be called an “empty account”, which then changes my first Then statement as such:
Scenario: Attempt withdrawal from an empty account WHEN a fixed amount of $100 is withdrawn from an empty account THEN the account remains empty AND an error message appears saying there are insufficient funds
Which do you prefer? Someone might argue: why would you say the account remains empty or remains at $0 at all? Wouldn’t that be somewhat obvious? Well, perhaps. But what if the bank allowed you to overdraw and put your account in a debt status, with a negative value indicating that you owe money for the withdrawal?
Let’s look at a few more exmaples, where you can make slight changes. In my current environment, somebody had proposed these two alternatives and wanted my opinion on which was “better”:
When a plan is created from a Phase 1 (Healthy Volunteers) study, Then it cannot be reforecast. When a plan created from a Phase 1 (Healthy Volunteers) study is selected, Then the reforecast options are disabled.
Again, those are two alternatives for the same test and the question was put to me as to what might be a good way to do this. I told the tester that, going with those examples, I actually liked a combination of both as such:
WHEN a plan is created from a Phase 1 (Healthy Volunteers) study THEN the reforecast options for that plan are disabled
That’s what jumped into my head initially. However, we started to talk about what “disabled” meant here. The tester’s instinct was not to state that options are disabled. I wanted to explore that a bit so I said we could do this:
WHEN a plan is created from a Phase 1 (Healthy Volunteers) study THEN that plan cannot be reforecasted
I told the tester that it was really up to us which of those WHEN/THEN groups we think is the better test. Both could be executed the same way, but the question was whether we conveyed the intent — along with enough of the implementation to understand what actions could and could not be taken. And this gets into another tricky area with these things: how much implementation detail do we provide?
I liked my first attempt simply because it stated a tangible: the reason a plan can’t be reforecasted is because the option to do so will specifically be disabled. (As opposed to, say, “not present.”) The “cannot be reforecasted” in the second revision potentially leaves open why the plan can’t be reforecasted. Is it because it’s not going to even be there as an option? Or it’s there as an option, but just not a usable one? Or will there be some text there saying as much to the user?
As yet another exercise in being flexible with these test specifications, I told the tester that we could even do this:
* A plan created from a Phase 1 (Healthy Volunteers) study will have disabled reforecast options
The asterisk there can stand in for a GIVEN, WHEN, or THEN. I use this approach when the GIVEN, WHEN, and THEN are all capable of being stated together as an overall test condition. If the preference ended up being to avoid saying “disabled”, then the above just becomes this:
* A plan created from a Phase 1 (Healthy Volunteers) study cannot be reforecasted
So this is just another in my series of continuing to figure out if BDD type concepts, like Given/When/Then, are effective and efficient for testers. I have so far come to the conclusion that they are. Specifically, that this kind of test specification writing forces testers to think more about what they are writing and what they really need the test to be saying — based, largely, on what the test specifically needs to be testing. I’ve also found that test specifications promote a certain fluidity of thought wherein testers have to more thoroughly engage in the subject matter of the application that they are testing.
