Test Specifications and Fluid Test Writing

I have been introducing Cucumber to testers who have little exposure to such tools. I was looking at whether The Cucumber Book would be worth having around the office. And while it may or may not be, one thing I notice is that it (like most resources on Cucumber I find) don’t really address some of the heuristics regarding how you can start thinking about writing test specifications.

I had talked about these concepts a little before when talking about specifying application workflow activities and writing tests as specifications. Likewise, I had talked about some general spec-writing guidelines. I want to delve more into those guidelines at some point.

Before doing that, however, I kept finding that all of this can be tricky for testers who are new to this stuff. In The Cucumber Book there is a distributed ATM example that is never really implemented but that is used as part of a multi-chapter running example. So to focus on something simple but also something specific, what I did was start with that example and break down some of the choices you have as you write test specifications. And I say that because I’m finding the cognitive hurdle a lot of testers have is the recognition of when to apply a given writing technique to a given situation based on a lack of understanding of the choices available.

So as an example, it seems clear to most people that the business rule for the ATM example is that the amount you withdraw will be subtracted from your account. So a fairly simple scenario is:

Scenario: Withdraw money from account

GIVEN I have $500 in my account
WHEN I withdraw a fixed amount of $100
THEN the balance of my account should be $400

That’s one way of writing it out. However, since the context (GIVEN) and the action (WHEN) can be wrapped up in a single, easy to read statement, I could say:

Scenario: Withdraw money from account

WHEN I withdraw a fixed amount of $100 from an account that has $500
THEN the balance of my account should be $400

If the first-person aspects are undesirable, it’s fine to do this:

Scenario: Withdraw money from account

WHEN a fixed amount of $100 is withdrawn from an account that has $500
THEN the balance of the account should be $400

It’s right here that I would tell testers to stop and think about the three variations above. They should ask themselves which they prefer. They should ask themselves which is more in line with how they think. It’s critical that testers new to this stop here and think about solutions like these. Getting used to the way you can morph the writing a bit is a critical skill to have when writing test specifications like this.

Okay, so now let’s consider how specific the test is. Do I even need to know that $500 is in the account for this? Not really, so I could say:

Scenario: Withdraw money from account

WHEN a fixed amount of $100 is withdrawn from an account that has sufficient funds
THEN the balance of the account should be $400

Hmm. But that doesn’t read well, does it? So in this case I may ask: do I even need the $100? Without knowing what “sufficient funds” are, a removal of $100 doesn’t tell me much and certainly doesn’t tell me that $400 is the end result. As such the $400 would also have to go to. What you end up with, essentially, is the business rule:

Scenario: Withdraw money from account

WHEN a fixed amount is withdrawn from an account that has sufficient funds
THEN the balance of the account should be reduced by the amount of money withdrawn

Again: here is a great place to stop. Encourage testers to think about what we just did here. My opinion is that this is a case where having the actual values may make sense. The specific example makes it clear even though the above is certainly accurate.

Assuming I put the actual numbers back in, does the example then overspecify at the price of clarity? Some testers might think that a specific example locks you in to just thinking about that example. Perhaps. For example, what the above scenario doesn’t tell you is what happens if you try to withdraw more than you have. Or when you try to withdraw when your account has no funds at all. Or when you try to withdraw with an invalid card. Let’s work out some variations.

Scenario: Attempt withdrawal using invalid card

WHEN a fixed amount of $100 is withdrawn from an account that has $500
BUT the card is invalid
THEN an error message appears

Now, going with our example from before, I could even combine the second clause:

Scenario: Attempt withdrawal using invalid card

WHEN an invalid card attempts to withdraw $100 from an account that has $500
THEN an error message appears

Now notice here that the $500 — unlike in our other example — probably is somewhat irrelevant. Because the the balance is not changing in this case. So I could replace “an account that has $500” with a valid business domain phrase. Maybe we’re back to this:

Scenario: Attempt withdrawal using invalid card

WHEN an invalid card attempts to withdraw $100 from an account that has sufficient funds
THEN an error message appears

So in that case do I even need the $100? Not really because, again, unlike the previous scenario, here the actual values used do not matter as much.

Scenario: Attempt withdrawal using invalid card

WHEN an invalid card attempts to withdraw money from an account that has sufficient funds
THEN an error message appears

Let’s stop again for a second. Here we have two different scenarios: one that we felt needed a specific example and one that we felt served as it’s own example without the need for specific data. For many testers I work with, this flexibility of design is actually what causes problem. Many testers seem to have grown up in a context where you have some designated guideline that tells you how a test should be written. What I just showed above is that you’ll allow yourself some flexibility on this, based on the nature of what you are testing.

Let’s continue with this last example. Is the information provided in the Then clause enough? Or should I be more specific? I could be specific about the nature of the error message:

Scenario: Attempt withdrawal using invalid card

WHEN an invalid card attempts to withdraw money from an account that has sufficient funds
THEN an error message appears saying to contact the bank

Or I could be specific about the content of the error message:

Scenario: Attempt withdrawal using invalid card

WHEN an invalid card attempts to withdraw money from an account that has sufficient funds
THEN an error message appears saying
  """
  The card you are using appears to be invalid. Your account has been
  made inactive.

  Please contact your bank, using the number on the back of your card,
  to resolve this issue.
  """

So it seems that what I’m doing is using specificity when it matters. As another example of that, let’s consider this other variation:

Scenario: Attempt withdrawal from an empty account

WHEN a fixed amount of $100 is withdrawn from an account that has $0
THEN the balance of the account should remain at $0
AND an error message appears saying there are insufficient funds

Note here “an account that has $0” could be called an “empty account”, which then changes my first Then statement as such:

Scenario: Attempt withdrawal from an empty account

WHEN a fixed amount of $100 is withdrawn from an empty account
THEN the account remains empty
AND an error message appears saying there are insufficient funds

Which do you prefer? Someone might argue: why would you say the account remains empty or remains at $0 at all? Wouldn’t that be somewhat obvious? Well, perhaps. But what if the bank allowed you to overdraw and put your account in a debt status, with a negative value indicating that you owe money for the withdrawal?

Let’s look at a few more exmaples, where you can make slight changes. In my current environment, somebody had proposed these two alternatives and wanted my opinion on which was “better”:

When a plan is created from a Phase 1 (Healthy Volunteers) study,
Then it cannot be reforecast.

When a plan created from a Phase 1 (Healthy Volunteers) study is selected,
Then the reforecast options are disabled.

Again, those are two alternatives for the same test and the question was put to me as to what might be a good way to do this. I told the tester that, going with those examples, I actually liked a combination of both as such:

WHEN a plan is created from a Phase 1 (Healthy Volunteers) study
THEN the reforecast options for that plan are disabled

That’s what jumped into my head initially. However, we started to talk about what “disabled” meant here. The tester’s instinct was not to state that options are disabled. I wanted to explore that a bit so I said we could do this:

WHEN a plan is created from a Phase 1 (Healthy Volunteers) study
THEN that plan cannot be reforecasted

I told the tester that it was really up to us which of those WHEN/THEN groups we think is the better test. Both could be executed the same way, but the question was whether we conveyed the intent — along with enough of the implementation to understand what actions could and could not be taken. And this gets into another tricky area with these things: how much implementation detail do we provide?

I liked my first attempt simply because it stated a tangible: the reason a plan can’t be reforecasted is because the option to do so will specifically be disabled. (As opposed to, say, “not present.”) The “cannot be reforecasted” in the second revision potentially leaves open why the plan can’t be reforecasted. Is it because it’s not going to even be there as an option? Or it’s there as an option, but just not a usable one? Or will there be some text there saying as much to the user?

As yet another exercise in being flexible with these test specifications, I told the tester that we could even do this:

* A plan created from a Phase 1 (Healthy Volunteers) study will have disabled reforecast options

The asterisk there can stand in for a GIVEN, WHEN, or THEN. I use this approach when the GIVEN, WHEN, and THEN are all capable of being stated together as an overall test condition. If the preference ended up being to avoid saying “disabled”, then the above just becomes this:

* A plan created from a Phase 1 (Healthy Volunteers) study cannot be reforecasted

So this is just another in my series of continuing to figure out if BDD type concepts, like Given/When/Then, are effective and efficient for testers. I have so far come to the conclusion that they are. Specifically, that this kind of test specification writing forces testers to think more about what they are writing and what they really need the test to be saying — based, largely, on what the test specifically needs to be testing. I’ve also found that test specifications promote a certain fluidity of thought wherein testers have to more thoroughly engage in the subject matter of the application that they are testing.

About Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.
This entry was posted in Communication, TDL. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *