For those of you who have read my posts on test specifications you know that I tend to focus on the idea of writing requirements as tests. This, to me, is essentially a large part of the benefit of collaboration-based approaches such as those suggested by Behavior-Driven Development (BDD) practices. (Although I still prefer Business-Driven Development, but whatever.) What I like about this approach, among other things, is that it builds in traceability. And I like that because requirements traceability was always something that I had an issue with as my career developed.

My problem is that the term “requirements traceability” gets tossed around quite a bit as if it were a solution to the problem of coverage. The problem for me is that a concept like traceability is not indicative of effective requirements coverage. In some places I’ve worked, the idea of being able to trace requirements to tests automatically equated to “covering” those requirements. All it did, in fact, was guarantee that some test could be traced to some requirement. It was always a statement of quantity, not quality.

Actually, I’m not even so concerned with traceability as a general rule as I am with how the requirements are viewed, particularly in terms of how — or even if — they are viewed as being distinct from the conditions and cases used to validate those requirements.

My personal view is that the farther apart statements of conditions and cases are from requirements, the more problems you are creating for yourself. I talk a little about this when describing conditions as part of a TDL.

The Collective Definition of Quality?

While I disagree with the Satisfice pundit James Bach on some things, I do like how he defined requirements (somewhat informally) as being “the set of ideas that collectively define quality for a particular product.” Certainly it’s the case that testing is often a process of assessing an agreed upon vision of product quality. With that in mind, it follows that if you have requirements, you can define testing relative to those requirements. Is that the same as testing relative to quality? Clearly the answer to that depends on the nature of the requirements but I think most people who have been in the industry longer than a year realize that the answer in no way depends on whether you have “requirements traceability.”

This really takes on a more refined focus when you realize that testing is not just an evaluative process. Testing also explores possible meanings relative to requirements because sometimes, with some projects, that’s all you can really do. Most experienced testers realize that many requirements are unstated in the sense that you often deal with a team’s shared understanding of quality, not all of which makes it into writing. That shared understanding has both objective and subjective components. Further, it’s often the case that technical constraints require modifications to requirements that have little do with business need but everything to do with implementing that need in working software. Because of those realities, an effective tester needs to be cognizant of unintentional gaps in the stated requirements and then try to resolve them to the degree justified by the risks of the situation. That’s really the key point: the risks.

Test Activities Becomes Associated With Requirements

When you perform testing, it’s done to mitigate risk, at least when you boil it down to the final essentials. Requirements will cover some of those risks, but it’s our test activities — associated with requirements — that will tell us what risk we are taking on. Going back to James Bach, I think he had a good point when he said this:

“The idea that a software product must satisfy its stated requirements is true if we define product quality as the extent to which we can reasonably claim that each stated requirement is a true statement about the product. But that depends on having a very clear and complete set of requirements. Otherwise, you’re locked in to a pretty thin idea of quality.”

What he means here is that while we can operationally define quality by defining the requirements, quality is most certainly not just some simple sum total of how many requirements were actually satisfied. That goes back to my point: just talking about requirements coverage is not only misleading, it’s simplistic — and that’s even more so if you end up tying your notion of traceability to coverage and use that as some indicator of “effective testing.” Not least of which, this view does not take into account that requirements have greater and lesser priorities. And, of course, some requirements will be in conflict.

I guess the point I want to get across here is that you can’t allow yourself to limit your thinking such that you consider requirements as disconnected conceptual ideas that are capable of independent evaluation without consideration of interdependence or even dependence upon the means of how those requirements will be tested. Those various means — the test activities — are rarely amenable to some one-to-one traceability.

Again, James Bach highlights well this kind of approach:

“To the extent that requirements matter, there should be an association between testing and requirements. But talk about traceability so often boils down to a form of clerical Bingo: ‘For each requirements ID, list the test case IDs that relate; for each test ID, list the requirement Ids that relate.’ The completeness of testing is then presumably evaluated by noting that at least one test is associated with each requirement. This is a pretty idea, yet I’ve seen projects where this checkbox traceability was achieved by defining a set of test cases consisting of the text of each requirement preceded by the word ‘verify.'”

I think what he’s saying here is that you have to explain the relationship between the test and the requirement: not just state that you can draw a connector between the two. Just saying we can connect or associate a requirement with a test is a simplistic statement. What’s important is being able to describe how that requirement and that test are associated.

One reason I like the requirements-as-tests approach is that it can force test writers to think about the requirements and not just do a rote connection between a putative test and a statement of requirement. It’s not just a matter of whether Requirement 1.2.3 is “testable” but rather of understanding what “testable” means for Requirement 1.2.3.

Quantity and Quality Measures

As I said earlier, one major part of the focus of testing is to bring risk to light, which is different than assuming testing exists merely to demonstrate conformance to stated requirements. The test process will be more persuasive if you can articulate and justify how test strategy relates to the definition of quality. This goes beyond just having “traceable requirements.” Okay, but what does all this actually mean? What does this talk of traceability and testability have to do with each other? My answer: the connection gets into the idea of quality and quantity in relation to testing.

Here I’m speaking to the notion of measuring quantity and quality in equal measures. To do that you need to agree upon the nomenclature in place. That said, agreeing to terms is not enough: you have to agree on how those terms are operationally employed within your overall development team.

Measuring a Quantity in Testing

Consider a hypothetical situation: two groups within one test team work on different projects for their manager. Let’s say each group is made up of two testers.

• The first group reports that they produced 37 test cases for about 100,000 lines of code in the past month.
• The second group reports that they produced 320 test cases for about 90,000 lines of code in the past month.

In trying to understand the reports, the manager discovers that the first group counted a test case as a script that sequentially exercised many software actions (requirements). The second group counted the exercising of a single action as one test case.

Both groups thought they were counting primitive elements and, to be sure, by their own definitions they were. The problem comes in because obviously their definitions were not the same. So a lesson to learn here is to provide a common definition of primitive units of measurement because then you have a scale from which to make comparisons and evaluations. Likewise, operationally define what test elements mean. Note that had both groups been tracing to requirements, the overall coverage matrix would have looked quite different.

Measuring Quality in Testing

Another hypothetical situation: a team testing 50,000 lines of code and 200 separate Web pages created 1,300 test cases, all traced back to requirements, and found 132 bugs. The bugs were removed and no failures appeared in the next test cycle.

So a question: did this result mean that there were no more bugs in the code or did it mean that the test cases missed some bugs?

The testers probably would not not be able to answer this question because they would have no way to measure test case quality. They would not know which actions, inputs, outputs, and parts of the code had or had not been exercised except in a most general way and yet — I hasten to add — they had fully traced their requirements. The lesson learned here is that traceability does not automatically equate to coverage. Coverage of different sorts can show different elements of how effective a test suite is.

So what do those things tell us?

The former situation might speak to how we trace our own test elements to our requirements specifications, particularly given that our specifications might leave elements out or refer to other documents as well as the fact that our specifications might have large “out of scope” sections.

The latter situation might tie in to the notion of the technique known as all-pairing: providing a basis for the combinations and/or permutations of test data that you plan to utilize when going through a test cycle. Further, making this manageable might tie into the notion of using a data generator and possibly even a test generator.

The point of both situations was really just to show that traceability, by itself, means nothing — just as coverage, by itself, means nothing. What matters is the notion of testability and that matters to the extent that it allows you to do an even more important traceability exercise: connect your quantity of work products to the quality of your work products.

My thoughts are still forming on some of these ideas, so what I sketched out here was a little rough. I’ll be revisiting this idea in future posts, particularly in terms of how these ideas relate to the effectiveness of techniques like TDL, approaches like requirements-as-tests, and principles like “testing is a design activity.”