In my post on Stories, Features, and Acceptance Tests I talked about a lot but I didn’t really cover the acceptance tests themselves. The tests are quite important. Keep in mind that the goal is to have specifications that are clear, sufficiently testable and falsifiable, and in line with client expectations prior to significant development or testing. In essence, the idea is to work out how a system would be tested as a way to check whether the requirements give you enough information to build the system in the first place. You shouldn’t dive straight away into how to implement something, but rather think about how the finished system will be used — i.e., acceptable use — and then double-check the requirements armed with that knowledge.
Acceptance tests should reflect the customers’ perception of when the application meets their requirements. This does not mean that such tests must be defined by a customer or business analyst but it does often mean that such tests are discussed with them.
But how do you write them? What do they look like? Well, let’s talk about that.
First, let’s cover a distinction some people trip over. What’s the difference between acceptance criteria and acceptance tests? You’ll get different answers depending on who you ask. Here’s how I like to think of it. Acceptance criteria refers to the set of acceptance tests. Put another way, all of the acceptance tests for a feature taken together are the acceptance criteria for that feature. Your features will probably be written up as user stories (as I covered in my previous post).
Important Point! A story’s behavior is simply its acceptance criteria.
If the system fulfills all the acceptance criteria, that means it’s behaving correctly. If the system doesn’t fulfill all the acceptance criteria, it’s not behaving correctly. How will you know the system is “behaving correctly”? You run the acceptance tests.
Important Point! Acceptance criteria are a set of passing test conditions that a given requirement must meet for it to be accepted as complete.
So given that, how do you write an acceptance test? Well, as with most things in the testing world, there is no one right way although there are plenty of people who will tell you their version of the “one right way.” Keep in mind an important point: acceptance tests are just tests. At a core level, what makes a good test? You have input, you have an action, and you have output. Acceptance tests are not really different.
Since acceptance tests are often derived as a result of conversation with various people, and since various people (not just testers) may ultimately write acceptance tests, certain structural motifs have been proposed. One such convention is known as Given/When/Then and it’s central to the notion of Behavior-Driven Development (BDD). The particular use of a convention — like Given/When/Then — connects the business-level concept of cause and effect to the application-level concept of input/process/output.
Let’s take a step back for a second. In my previous post, I mentioned different “templates” can used to capture a user story. The example I used was:
As a [user role],
I need [the ability to something],
so that I can [get some benefit or avoid some consequence].
That structural motif is known as the Connextra format. As I mentioned, the above format gives you a business need. With your tests, you basically want to translate the customer business need so that what they really in effect said to you was a statement like this:
Given [some initial context],
when [this certain type of event occurs],
then ensure [this particular outcome].
What that boils down to, as a template, is this:
Given [input | preconditions],
when [actions | triggers],
then [output | consequences].
The template is loose enough that it shouldn’t feel artificial or constraining to analysts (or customers) but structured enough that it can break the story into its constituent fragments and provide an observable test for the business need. It’s important not to lose sight of what you’re doing here, which is describing the acceptance criteria in terms of specific test conditions. In fact, it’s probably a good idea to see how the two templates (or statement patterns) I’ve talked about really do say the same thing but at different levels.
| User Story | Acceptance Test |
| As a [user role], | Given [input or preconditions], |
| I want to [be able to do something], | when [actions or triggers], |
| so I can [get some benefit or avoid some consequence]. | then [output or consequences]. |
So let’s go back to the example I provided in the previous post:
Story:
As a project manager user
I need the ability to manage tasks
so that I can make effective decisions regarding work being done.
Feature: Ability to Add Tasks
Scenario: Add a valid task.
Scenario: Add an invalid task (missing required fields)
Scenario: Add an invalid task (date already passed)
Scenario: Add an invalid task (associated to an inactive entity)
Feature: Ability to Filter Tasks
Scenario: Filter on "show active and overdue tasks"
Scenario: Filter on "show overdue tasks only"
Scenario: Filter on "show completed tasks only"
This is the overall story. It’s the conversation. Obviously it’s not as detailed as a functional specification. You can argue that this is the start of the acceptance criteria since the customer is saying that in order for them to accept the system, it must have the above features. The scenarios help flesh out the conversation a bit but, by themselves, they are not tests.
So now let’s write some acceptance tests for one of the features.
Scenario: Filter on "show active and overdue tasks"
Given I am a local admin
Given I am viewing a contact that has all task types:
completed active task
completed overdue task
non-completed active task
non-completed overdue task
When I select "show active and overdue tasks"
Then I should not see the completed active task and the completed overdue task
And I should see the non-completed active task and the non-completed overdue task.
Scenario: Filter on "show overdue tasks only"
Given I have a tasks page for a contact that has the following:
When I select "show overdue tasks only"
Then I should not see the following:
And I should see the following:
non-completed overdue task
Scenario: Filter on "show completed tasks only"
Given I have a tasks page for a contact that has the following:
completed active task
completed overdue task
non-completed active task
non-completed overdue task
When I select "show completed tasks only"
Then I should see the following:
completed active task
completed overdue task
And I should not see the following:
non-completed active task
non-completed overdue task
So what I did here is I went from the conversation and got to the confirmation. This took the form of acceptance tests, which essentially further break out the acceptance criteria. Only when we get to this point, do we actually have a requirement. Ideally other than minor text alterations, no further changes are allowed to these requirements until the story is delivered.
Let’s take a slightly simpler example. Let’s start with a simple statement:
Users can log in to the system.
That’s the start of a conversation. But login is pretty simple, right? Of course someone needs to login. So how about this instead:
Invalid users should not be able to log in to the system.
Here we have a simple statement that’s probably workable by itself but let’s turn this into a user story (business need):
As a [system administrator]
I need the application to prevent invalid logins
so that client information is protected.
We have the start of our acceptance criteria. Now let’s turn that into an acceptance test.
GIVEN a user logging in to the application,
WHEN the user attempts to login (with username ‘jnyman’ and password ‘1234test’),
THEN the user will not be logged on. (Error message: ‘invalid password’.)
This now is a further elaboration of the acceptance criteria as an executable statement. This specific test serves as an example.
There may be other reasons why an invalid login occurs. I could spell out each one in a different Given/When/Then. Or I could generalize the above and then provide an example table. Here’s an example of that:
GIVEN a user logging in to the application,WHEN the user attempts to login with username and password
AND an error condition occurs,
THEN the user will not be logged on
AND will receive notification of the error condition.
EXAMPLES
| Error Condition | Username | Password | Notification |
| Invalid password | jnyman | 1234test | ‘Invalid Password’ |
| Invalid username | jeffnyman | test1234 | ‘Invalid User’ |
| Service down | jnyman | test1234 | ‘Application Service Unreachable’ |
| Database down | jnyman | test1234 | ‘Database could not be accessed’ |
This is just one example. It’s what you might call a generalized example-driven format. I say that because the Given/When/Then outlines the general test condition for this feature — i.e., how it’s going to be confirmed. (Although note that these conditions don’t say how the implementation will take place.) From these test conditions, test cases can be crafted.
So what you should get out of what you’ve seen so far is that you capture elements in both the conversation and the confirmation, but it’s the latter that makes the requirement operational. The operational aspects of the requirement can be captured as tests of that requirement.
Important Point! A story encapsulates a business need. A test encapsulates the business understanding.
What I hope this post has done is show you one possible structure for what acceptance tests look like. One of the main things that can come out of this approach towards gathering acceptance criteria is treating “example usage” (user stories, in some cases) as the tests. This is known by some people as example-driven testing. If you want to call it that, feel free. What’s happening is that you’re asking someone for an example of how they would like to see some aspect of the system operate. The answer you get is encoded as an executable example, a test.
The key feature of the examples is that they should provide enough information for developers to implement and for testers to verify the stories planned for the given release. In fact, those examples should build the test verification part in because they should serve as acceptance tests. Those acceptance tests can then serve as drivers for the design and development that must take place.
Think about what the examples are doing. If crafted effectively, they are providing a set of actionable examples of how the customer or business expects the application to behave. That’s a really important point because what you ultimately test for is behavior. Put yourself in the customers’ or business’ shoes. What they’re really saying as a result of these examples is:
I will accept the application, if the application behaves this way.
And what that really translates to is:
I will accept the application, if these examples can be executed correctly.
And since I just said the examples are acceptance tests, what’s really being said is this:
I will accept the application, if these tests pass.
Different wording, but the same thing is being said. In the end, it really does come down to tests — at least as far as I’m concerned. An acceptance-driven approach is basically where tests have been built-in to the design process, not just at the unit (code implementation) level but at the application (business intent) level. The tests are now part of the requirements.
In fact, the tests really are the requirements.
As if that wasn’t exciting enough, those examples have now told the developers exactly what is going to be tested for. So what the developers have to do is design a solution that will pass those tests. And how do they know that their solution will be valid? Because the customers/business have already told you so. You’ve captured their knowledge as tests and they’re going to design to those tests.
What I hope you got out of this, however, was that telling a story is a very effective way to get the information you need to find out what your users want. The process of creating acceptance criteria allows you to figure out how to prove that you delivered what they wanted. The process of creating acceptance tests makes sure that everyone is focused on providing a way for you to execute those proofs.
