Constraints and Cost of Mistake Curves

In this post I want to explore how a theory of constraints can be combined with cost of mistake curves to consider how testing operates, first and foremost, around the concept of design. Keeping design cheap at all times is a value of testing that I rarely see articulated. So here I’ll give that articulation a shot.

Testing has to be performed any time we are likely to make mistakes. Further, it has to be performed as close as possible to those times when mistakes are made. So there’s an obvious question: when we do we make the most mistakes?

I doubt it’s too controversial to say that we make the most mistakes at the following two points:

  • When we talk about what to build. (developing requirements)
  • When we build what we talked about. (developing features)

It can be instructive to think in terms of a theory of constraint around testing.

Theory of Constraints

A theory of constraints is often about identifying the most important limiting factor in your environment. That limiting factor is usually stated in terms of a goal, such as “getting software out the door quicker” or “producing software with less bugs.”

The Bug Constraint

Many times we talk about the cost of bugs being that limiting force. Consider an oft-quoted remark by Mark Zuckerberg:

“…developers, moving quickly was so important that we were even willing to tolerate a few bugs in order to do it. What we realized over time is that it wasn’t helping us to move faster because we had to slow down to fix these bugs and it wasn’t improving our speed.”

But what if you drive the cost of a bug as close to zero as possible? Then it doesn’t matter how many bugs you create. When the costs are low, you can experiment more and make more mistakes.

This is how a cost of change curve becomes a cost of mistake curve. With this viewpoint in place, it’s not the cost of change that matters; it’s the cost of the mistake. It’s not how late in the cycle we try to change something. Instead it’s the time between when we create the problem and find it. The longer that is, the more expensive it is.

If you close the gap between creating and catching the mistakes, less change occurs between the two events.

To do all that we need feedback. But feedback itself is subject to a theory of constraints.

The Feedback Constraint

Consider that tests are a means of feedback to figure out whether we made mistakes. So the longer it takes us to get feedback from tests as to whether we did or did not make a mistake, the more expensive we are making our process. Thus the less value we get from that feedback. This means every mistake gets more expensive.

That said, the problem with tests is often not even just slow feedback; it’s that eventually they give us no feedback or limited feedback. We stop running them; or we only run subsets of them; or we have tests at so many abstraction layers that the feedback we are getting becomes muddy.

This often means you are finding the more obvious mistakes at the cost of not finding the more subtle, but often more detrimental, mistakes. Those mistakes that are, for example, driving up your technical debt. Or reducing your ability to scale or respond to various market or technology pressures.

In that case, an interesting thing happens. You end up driving down the cost of mistake curve for mistakes that aren’t that expensive (and thus not as important) but you also drive up the cost of mistake curve for mistakes that actually are expensive (and thus are important).

Even if you do create a fast feedback loop, you have to make sure you produce the right feedback.

But what does the “right” feedback mean?

Well, consider that you can get no feedback at all. Or you can get feedback but do so poorly. You can only get some types of feedback and not others. You can be getting feedback around aspects of work that lack of demonstrable value such that any feedback you get is compromised.

Your feedback mechanism may not allow you to make better or any decisions at all. Your feedback mechanism might provide a bias that allows you to make wrong decisions.

But how does this tie back into tests?

Putting Pressure on Mistake-Prone Areas

If tests are used to put pressure on both areas where we make the most mistakes, we stand a good chance of finding the simplest solutions that provide the right amount of value. This allows us to avoid building that which need not be built and avoid adding too much complexity to what does need to be built.

This helps us with those feedback issues we just talked about.

But that introduces another constraint: time.

The Time Constraint

To get feedback from tests takes time. Not just to write those tests, but to execute them.

To be sure, every decision related to development costs time. And that cost of time is distributed, which can sometimes make it difficult to reason about what exactly caused the inflation of time.

What people on a delivery team often want to know is how long it will take to change a feature. A key metric here should be the cost per feature over the time interval that matters.

To change a feature means to change the code. And how hard it is to change the code can be directly measured by the quality of that code. And that quality can be stated in terms of how testable the code is. Or, rather, how much testability is and was a primary quality attribute that the delivery team put emphasis on.

A change in code triggers a change in behavior which in turn is reflected in how the feature works. We want warnings that such changes have taken place. Tests are such warnings.

This is how you relate the value of tests relative to the time it takes to create them. Tests can add a great deal of value if they encourage a design style that makes changing features easier which means making code changes easier. A key metric here is how easy it is to find incorrect behavior and add new behavior.

This gets into design. Which is its own kind of constraint.

The Design Constraint

There are a lot of practical aspects of design but, conceptually, it is arguably the case that design is navigating the decision points that have consequences for the future. The purpose of design is ultimately to reduce the cost of change.

But to reduce the cost of change, we have to reduce the cost of mistake.

The Role of Specialized Testing

Whether we’re “adopting DevOs” or “being agile” or “shifting left,” what really matters is if testing, as a specialized discipline, is enabling your delivery team to make better decisions sooner and thus deliver demonstrable value to users at a pace that is sustainable, both for the business and for the users who derive value from the features the business provides.

Part of what testing does is put pressure on design, ideally helping to keep design cheap at all points. Design has to be kept cheap as much as possible because the quality of our design will have to provide a means of delivering quality.

But “delivering quality” can be tricky. Confidence in value manifests as a perception of quality. Yet, quality is a shifting perception of value over time. That perception is often one that occurs in the context of a specific design.

Design Evolves

As a tester you learn how to put pressure on design at least on two levels: at the business level and at the code level. The latter helps you discuss code with developers. The former can help discuss features with business. It’s the intersection of those two areas — business and code — where testing becomes a very diverse discipline.

Two big challenges in this area where code and business intersect are keeping the architecture in sync:

  • First, with the requirements as they are uncovered or evolve
  • Second, with the implementation as it gets built and evolves

This means our delivery team needs to design testable requirements. And our delivery team needs to design testable code.

Tests Drive Everything

The entire delivery team has to start building features by thinking about how to test them. We think about how to test them by thinking about what our customers need to be successful. We think about what is value-adding to our customers. Then we build that.

Tests guide the requirements and they guide the code. At minimum, tests specify what we want a successful interaction with the feature to look like. Then we use the test to design that interaction.

All of that is what specialized testing is about.

But to get to that point, in our current industry, I believe that you have to consider the theory of constraints that testing is operating within and you do have to consider how testing is enabling an effective cost of mistake curve for the delivery team.


This article was written by Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.