I think the future of testing relies less on a sole focus on test execution and more on the ability to write tests in the language of the business domain, effectively tying together the role of business analyst and tester. I think that future also includes the ability of testers to instrument certain artifacts, like requirements documents, so that the requirements, acting as tests, can be converted to different formats, most notably automated tests. There’s a lot of contention out there around these ideas from “purist” testers: those who feel that these attempts provide a single-source test format is getting away from testing and more into development.

While I agree in part that this is putting some focus on development activities, I disagree that this gets away from testing. Effective testing, to me, is about putting testing where it will do the most good in the most responsible fashion. Sometimes that testing is finding new bugs, sometimes it’s providing confidence that bugs haven’t come back. Either way, testing is all about communication.

I think we’re at the point where testing has a primary goal of business acceptance, where the end result is that testing is used to improve communication, show mutual understanding, and provide mental alignment. What this leads to is a breaking down of the traditional boundaries around testing and the requirements specification processes as well as how and to what extent development activities take place in relation to testing activities.

I believe that testing is a design activity. Tests can be an input to the design of a system. This is what aligns testing with traditional business analyst roles. The core idea of business acceptance is to build a shared and consistent understanding of the domain. This is done via the use of an effective and operational shared language.

An effective approach to this goal can be to write requirements — and thus acceptance criteria — in a Test Description Language (TDL). Any TDL should provide motifs. For example, FIT/FitNesse and Robot Framework provides tables. RSpec provides the describe/it notation. SpecFlow, JBehave and Cucumber provide a Given/When/Then structure. These structures can provide you with different ways to encode the information, such as by allowing you to specify the test data information separately from the narrative of the test.

A good test script, whether considered from an automated or manual perspective, should be workable as a set of examples of correct behavior. A set of related examples should be capable of being dumped into some structure. In FIT/FitNesse or the Robot Framework, this would be a separate table. In Concordion this would be a section of HTML. In RSpec this would be a describe block. In SpecFlow, JBehave or Cucumber this would be a scenario. This structure can be encoded as part of an automation framework. The framework generalizes the details of how to execute a test script, leaving the test script — again, manual or automated — to say what should be tested.

The point is that with enough formality, tools can be written that do two very specific things:

1. Interpret the intent of the requirement and then drive the application under test to ensure that the requirement works as stated.
2. Break out the condition level wording (of what should be tested) into step-by-step instructions (indicating how it should be tested).

I believe this is the future of testing tools. But I think it’s important to see a basis for this approach. Note I say “a” basis and not the basis. I think testers come to this approach in their own ways. Here I’ll show you a little bit of mine.

“I want a car. But not that car.”

Way back in the Dark Ages — so around 1999 or so — I read Alan Cooper’s book The Inmates are Running the Asylum. One particular aspect of the book really stuck with me. Essentially it boils down to imagining a requirements specification where your company is designing a product for a new customer base. So let’s say we have these requirements:

1.0 The product must have a gasoline-powered engine.
2.0 The product must have four wheels.
      2.1 The product must have a rubber tire mounted to each wheel.
3.0 The product must have a steering wheel.
4.0 The product must have a steel body.

Okay, so what’s the first thing that comes to your mind?

Obviously many people would suspect a car. Most any modern car would satisfy the requirements given. (Although with the emergence of the “green car” market, I suppose the 1.0 requirement might not be so solid.) Now, while any car may suffice, I may in particular be thinking of a 1996 hunter green Nissan Pathfinder. But maybe you’re thinking of a 2009 tuscan beige Acura RL.

I guess we can just hope those differences come out in the requirements. Then again, do all those differences even matter? Maybe we should meet to discuss whether make, model, year, color, and so on all impact the design of the car for our purposes. After all, my Nissan and your Acura are both going to have four wheels and a streering wheel, right?

But … WAIT!
What if we all totally missed the point?

Let’s say we ask the customers what their goals are for the product we are creating for them. And they say something like this:

Well, the end result should make it easy and fast for people to mow their lawn.

Whoops.

So by understanding what the end-user actually wants to the use product for and why they want it, you just got a totally different view of the product. Obviously, right? After all, they’re talking about a riding lawn mower, not a car. Notice here that with this statement of user goals, you’d still have to flesh out some concepts further. For example, what does it mean to say “easy”? What does “fast” mean in this case? And you’ll probably get something like this as well:

Oh, and they should be comfortable while using it.

What does “comfortable” mean? We may think we know what is meant by those terms. But, then again, a few seconds ago we were also pretty sure we had to design a car and not a lawn mower. Lesson (hopefully) learned.

Here’s What I Need … And Here’s How I Want It

Another good example, but in a different context, comes from Mike Cohn’s 2004 book User Stories Applied. The focus this time is on making sure that design decisions, or implementation details, are kept out of the stories as much as possible.

Let’s say the basic requirement (“story”) is that the user can compose and send an e-mail message. But now let’s say that this requirement is fleshed out like this:

User selects a ‘New Message’ menu item.
The application will pop-up a ‘Compose New Message’ window.
The user can edit the e-mail body, subject field, and recipient text fields.
The user will then click a ‘Send’ button and the application will send the message.

Clearly there are assumptions about the design of the user interface throughout this requirement. In fact, you might note that this “requirement” is really a low-level functional test. If you have a good analyst, they should have been asking a few questions as they read the above “requirement.” Namely:

• How relevant is this requirement if there is no ‘New Message’ menu item?
• What if there’s some other way to get a ‘Compose New Message’ window?
• And, speaking of that, how do we know that there will be a ‘pop-up’ window in particular?
• Do we know that the recipient will be a text field? (What if we have an interface that has a list view of all possible recipients?)
• What if there is need for a voice-recognition system as a possible interface?

You can see there’s a lot of assumptions wrapped up in that implementation.

So what would be an actual requirement here?

At its simplest, it’s really a test condition on the system:

A user can compose and send e-mail messages.

That serves as a basic, high-level requirement or acceptance criteria: the system must support the idea (or scenario) of a user composing and sending an e-mail. That’s the intent of the system. The details of how that can happen (i.e., under what conditions or specific scenarios an e-mail can be composed and sent) can be fleshed out as acceptance tests. That’s the implementation of the system. Any design that fulfills the above criteria — and that passes the acceptance tests derived from it — can be considered valid. What proposed design the customer wants is, of course, up to them.

There are some key words and phrases I brought up so far.

• I brought up a distinction between “intent” vs. “implementation.” A what vs. a how. That implies a distinction between design work and coding work.
• I brought up the idea of a requirement and acceptance criteria and I seemed to conflate the two.
• I brought up the idea of “tests” and I put the word acceptance in front of that.
• I brought up the term “case” and used it in the context of “intent.”
• I brought up the term “condition” and used it in the context of “implementation.”
• I brought up the term “scenario” and used it with pretty much all the other terms.

I think those are important concepts to consider because you will be using those terms and how you use them will matter. (See Testing’s Brave New World where I at least tried to give some focus to my own no doubt muddled thinking.)

“I’m thinking of a shape …”

Another book I was reading was called Speaking in Styles: Fundamentals of CSS for Web Designers. The book asks you to think of a shape. It can be any shape you want, as long as it is a bounded shape. So that can be a circle, a square, a triangle, a parallelogram, whatever. It just has to be a closed shape.

Got one in mind?

Now, with that shape in mind, the book asks you to think about how you would describe the shape to someone. Here’s the catch, though. You have to do this without talking about any shape. You can use the term “line” since that’s not a shape but you can’t say something like “Okay, now have a curve that’s sort of like a circle” or “connect the lines together at an angle like they were part of a triangle.”

Are you able to provide the description in such a way that someone can recreate the shape you meant? Try it. In fact, this is a fun exercise sto try sometimes. Describe the shape you have in mind to another person and have them try to draw what you are describing.

The book then settles into a simple shape by way of example: a square. I’ll do the same here. Here’s a possible way to describe the square:

“Draw four lines. Each of the lines should be the same length. The lines must all touch each other. Two of the lines will touch horizontally and two of them will touch vertically.”

How accurate does this sound to you? Do you think it’s practically inevitable that anyone hearing this would draw a square like this:

Clearly we have four lines. All lines appear to be the same length. We have two lines (the top and bottom) touching horizontally and two lines (the right and left) touching those lines vertically. Do all the lines touch? Hmmm. Well, yeah. Sorta. Although not all lines touch all lines. For example, right and left (vertical) lines never touch each other. Likewise, the top and bottom (horizontal) lines don’t touch each other. Does that matter?

The answer to that last question can make a huge difference, particularly based on the wording. Someone could just as easily draw something entirely different. For example, here’s one interpretation:

Notice how we seem to have fulfilled the requirements but we have not drawn a square at all. But that interpretation has the same problem as the other. The original description said “The lines must all touch each other.” So maybe this interpretation could be generated:

So what about those last two interpretations? You might be thinking: “Well, geez, what idiot would have those interpretations of what I said?” Yet the problem isn’t with how smart the people are or how well they listen. The root problem is that humans make a lot of assumptions when it comes to communication.

When we communicate, we’re not just following the stated directions; we’re also calculating what the person giving the directions is actually thinking. We fill in missing details, such as that all four lines should touch at their ends. That was never specified. That’s why the second interpretation above was a valid interpretation. There was ambiguity in whether all lines had to touch each other or just that all lines had to connect up in some way. That’s why the third interpretation could be valid.

Also keep in mind that while you knew I was talking about a shape, if you tried this exercise with someone who didn’t know that, their mind may not automatically be thinking of this in terms of shapes. This shows how a certain context or bit of knowledge can allow you to make certain safe assumptions that others who lack that context simply could not make. Yet, when communicating, it’s quite easy to fall into the trap of just assuming other people must somehow have the same context as you.

All of this just goes to show that the intent and the implementation can be wildly diverging things. You’ve just seen examples of that in a few contexts.

The Speaking with Styles book brings up a really interesting point:

Think about this for a second in relation to design work. As good as they are becoming at understanding humans (via different domain-specific languages), technology tools still require that we communicate explicitly when we describe something, especially when that something is a design, whether visual or via narrative. A tool will (usually) not fill in the details. What is needed is a language that you can use to quickly and precisely describe your designs in such a way that the tool will understand them without fault.

The idea of communicating explicitly with a specific language is what coming up with behavior-driven acceptance tests — or doing acceptance-driven design — is all about. Written requirements are often subject to the same level of interpretation that the above “instructions for a square” are. In fact, it’s even worse because of the inherent complexity of most applications. We need a way for people to communicate explicitly when they describe some needed feature. We can’t always count on developers or testers to “fill in the details” since the details we choose may not be what the customer actually wants. As with the design above, what’s needed is a “language” that we can use to quickly and precisely describe the requirements — which are a form of design — in such a way that developers, testers, and the customer will understand them with a minimum chance for interpretation error.

Taking a specific example in terms of Web design, one such language is called Cascading Style Sheets (CSS). Why I’m choosing this particular technology to make my case will become clear as I go on but let’s stick with the CSS idea for now. Many people eventually find that “speaking CSS” is not that different from speaking in English or any other human language althouth it does, as expected, take a little adjustment to get used to its particular syntax (how things are said), semantics (what things mean), and vocabulary (what to say). The same applies to behavior-driven structures for acceptance tests. By way of example, consider this Web design statement in English:

“The border is solid red and 5 pixels thick; the background is pink; there is a 25 pixel padding around the content; the font is Arial at 8 pixels; and the text is double spaced.”

The same description in CSS isn’t actually all that different:

border: 5px solid red;
background-color: pink;
padding: 25px;
font: 8px/2 arial;

Notice that most of the words (vocabulary) are almost identical. It’s primarily the structure (syntax) that’s different, but it’s really not that hard to decipher. The point of this example is that you can describe your requirements in behavior-driven structures as easily as you do in English. This is largely the same logic by which Web designers learn to work as easily in CSS as they do in English.

All of this just goes to show that the intent and the implementation can be brought together. You’ve just seen examples of that in a few contexts.

Okay, but how can this work in terms of testing and requirements? Well, right now I’m in the middle of a large project where I’m trying to decide once and for all what my thoughts are on tools like Cucumber and SpecFlow, where you write tests in a focused way, with particular structural motifs (like Given-When-Then), as well as the general idea of instrumented documents, like the Wiki pages of FitNesse or the HTML pages of Concordion.

I’ll be posting about these experiences as time goes on. As you can probably guess, this post along with Testing’s Brave New World is really my attempt to bring a lot of concepts together so that they can be discussed in a particular context. That context is the idea of testing as a design activity, that serves as an input to development activities. Testing is infused within the entire development process, from product inception down to product deployment. These ideas are not new necessarily; but I think there is so much out there for testers, analysts, and developers to focus on that it actually becomes possible to defocus activities.