Testing That is Effective, Efficient, and Elegant

There is notion in quality assurance and testing between verification and validation. Verification asks “Are we building the product right?” Validation asks “Are we building the right product?” Some people use this very distinction to draw a line between the activities of quality assurance and the activities of testing.

My own view is that using terms like verification and validation may draw up a false dichotomy. I say that because for me testing is a design activity and, as such, has a broad mandate. I believe that tests can be a shared language. I believe that tests can directly encode specifications and requirements. I believe that tests can act as a domain knowledge resource. As such, I believe too much focus on a distinction between “verification” and “validation” is an ineffective and inefficient way to recognize how testing complements quality assurance.

When testing and quality assurance are not effectively seen as two distinct things where one complements the other, then I also believe any approach to either lacks a certain amount of elegance.

Now, all this being said, I acutally do think the distinction of verification and validation stands on some firm ground. After all, efficiency refers to doing things in a right way. Effectiveness, on the other hand, refers to doing the right things. Efficiency tends to focus on the process (i.e., the means of getting to an end) whereas effectiveness tends to focuses on the product (i..e, the end that is reached). Another way to put it is that efficiency tends to focus on avoiding mistakes in the first place whereas effectiveness is about achieving success, regardless of the mistakes encountered along the way.

But I do think being effective and being efficient can be a more operational way to look at things than simply talking about verification and validation. I say this because testing absolutely demands flexibility and innovation.

And here’s where an interesting paradox may raise its head. In order to be efficient repeatedly, a certain amount of discipline and rigor is required. This very discipline and rigor, however, can build inflexibility into the whole affair. Doing the same thing again and again in the same manner can discourage innovation.

Since efficiency is linked to verification in what I’m talking about, this means that verification can become almost too much of a static sort of affair.

Effectiveness, on the other hand, tends to be more adaptable to a changing environment and thus requires flexibility. To be effective tends to encourage innovation because it demands that people think about what they are doing at the moment and ways in which to do it given the reality of current circumstances. Yet this very flexibility and adaptability lead to a profusion of choices and a potential inability to have a cohesive effort that is repeatable beyond its current circumstances.

Since effectiveness is linked to validation in what I’m talking about, this means that validation can become almost too much of a dynamic sort of affair.

This is where I think elegance steps in. It gives you a “tie-breaker”, as it were. Or a way to view your effectiveness and efficiency through a different set of eyes. The elegance does not just boil down to “did we do it better?” or “did we think about it better?” but rather to “do we understand it better?” or “have we built a base for the future?”

One way to look at elegance is in terms of the minimization of complexity. In the development world, you’ll hear some people break the complexity of a solution down into that of essential complexity and accidental complexity. So for testing solutions to be elegant, I believe you will find they are made up only of essential complexity and little to no accidental complexity. Sound perilously vague? Well, since many of us will disagree on what is “complex,” it probably is vague. To me the complexity of testing solutions comes in when choice DOES NOT exist in the system and too much uncertainty DOES exist in the system.

If you allow for innovation and flexibility (effectiveness) in testing, but maintain a certain discipline and rigor (efficiency), then there have to be some guidelines in place for how to handle choice (in terms of providing it) and uncertainty (in terms of removing it).

So rather than droning on about this, how about if I just give some examples of what I mean? Specifically I want to show how I try to get test teams to think in terms of efficient, effective, and elegant. I like to start out with some “axioms” for lack of a better term. I try to keep these relatively short and focused. These are the things that we, as a team, have to believe and act like we believe. The first point, in fact, is a statement of what I just said!

  • Testing can be carried out effectively, efficiently, and elegantly.
  • Testing involves an active, skilled, technical investigation.
  • The goal of testing is ultimately the persuasive communication of relevant information in a timely manner.
  • Testers are writers and editors. Thus an ethic of elegance and pride is required to produce graceful work that is motivating and stands up to scrutiny.

Now here are some ways I might try to bring this stuff home in terms of examples of what I mean. First let’s consider these concepts in relation to a test.

  • An efficient test should focus on input, process, output.
  • An effective test should be expressive and reveal intent.
  • An efficient test should focus on one observable action result at a time.
  • An effective test will group related observables.
  • An efficient test provides a specific example of a data condition.
  • An effective test provides a generic data condition that is being tested.
  • An elegant test should specify behavior and describe functionality.

Now let’s consider these concepts in relation to a test set (however you want to define that).

  • An efficient test set specifies valid and invalid input data conditions.
  • An efficient test set checks for valid and invalid output data conditions.
  • An effective test set group data conditions into classes of observables.
  • An elegant test set can be composed to explore business rules and workflows.

Finally, let’s consider these concepts in relation to testing as a practice.

  • Efficient testing use scripts that provide a structure to frame exploration.
  • Effective testing use exploratory methods that add variation to scripts.
  • Elegant testing uses scripted exploration to show how customers value the application by how it is used.
  • Efficient testing uses scenarios that show how the product performs a capability.
  • Effective testing uses scenarios that show what has to happen to meet a need.
  • Elegant testing describes requirements and demonstrates features.

It really is important to consider these things because what you do and how you do it now will provide the basis for what and how you do things in the future. It also keeps a team dynamic and thinking about concepts. Here are some examples of what I mean by that:

  • Some testers prefer the term test conditions to test cases. Others prefer the reverse. Some prefer not to use two terms at all. How I promote this is that efficient testing groups related test conditions as variations on a test case. Effective testing uses test conditions that are predicated on specific data conditions.
  • Terms like “positive”/”negative” testing went out of style with experienced testers awhile ago. Elegant testing ultimately focuses on stating valid and invalid conditions. This means testers must apply efficient and effective testing such that they identify all the things that can be varied during a test (leading to both valid and invalid data conditions) and make sure they make smart decisions when we choose specific variations and exclude other variations.
  • Elegant tests are ultimately “best of breed.” In a group of tests that have a similar intent, time and resource limitations may mitigate toward the execution of only a subset of these tests. In such cases, the test that has the highest likelihood of uncovering a whole class of errors should be used. Such tests can be very elegant.
  • An efficient test should be neither too simple nor too complex. Although it’s possible to combine a series of tests into one test case, the possible side effects associated with this approach may mask errors. This would imply that an effecetive test has a different observable (or a different path to the same observable) and should be executed separately and considered a separate test.
  • Some test techniques are efficient in terms of choosing among specific inputs and arranging those inputs in combinations or in sequences. An aspect of elegance creeps in when testers apply decisions concerning feature interaction, data flows, and choosing paths through the user interface that encode understanding about how users actually use the system.
  • An effective test case must be capable of revealing information. You want tests that will give you the type of information that you are seeking. The goal is not necessarily to find a bug; the goal is to gather information. The value of a test is not in whether or not it can find a bug, but in whether it can provide the information you need. (It just so happens that often the information we need would reveal a bug if something was wrong in the application.) Elegant tests put a preimum on revealing intent.
  • Effective testing means we must understand requirements as they relate to how our users value the product we are testing. This means we have to understand users and not just read specifications or requirement documents. Elegant testing will use heuristics to structure and guide that process of encoding that understanding. Elegant testing uses tests to tell a compelling story that shows the actions of users.

I probably should have noted at the start of this post that I really had no intention of dictating what is and is not effective, efficient, and elegant when it comes to testing. But what I did want to show is that I believe test teams that make these distinctions stand a good chance of thinking and seeing differently. What I put here was obviously just a series of my own opinions — those that I actively work to promote.


This article was written by Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.