I was going to title this post “Things That Matter” but that’s a bit too vague. Further, in the previous post on product acceptance I talked about one aspect that falls out of that acceptance-style thinking, which has to do with metrics that matter. So let’s talk about that a bit.

Here’s a question that delivery teams should ask: “What matters to us?” It sounds like a simple question but, from a quality context, it’s one that collectively we often don’t answer very well or even at all.

To know which questions are the right questions to be asking, you have to know what’s important and what matters. And you can’t know that unless you know which outcomes are desirable. Thus all “things that matter” should first come from the desired outcomes. And that certainly applies to metrics.

Core Metrics

I’ll start by noting that awhile back I wrote about cautionary tales on metrics. With that in mind, and speaking from the context of the world I tend to live in, I personally believe that all metrics (outcomes) that truly matter for a delivery team are these:

• Feature coverage
• Feature readiness
• Production readiness
• Production stability

These are the metrics that matter whether you’re Agile, Scaled Agile, Agile-fall, DevOps, etc. It doesn’t matter. Any other metrics you may want to track can fall out of those. Thus those are “things that matter.”

Remember how I said the original title was too vague? And yet, as you’ll see, it’s really going to turn out to be the more accurate title. Because in the end I do believe metrics are secondary. They have a purpose and a useful one, but they can’t be the ends in themselves.

What’s interesting about considering metrics in relation to these “things that matter” is that it leads to other things that matter. Let’s see how this works.

Statements About Metrics

Consider this statement about metrics:

• One of the more effective software metrics is cost per feature over the time interval that matters.

Agree? Disagree? No opinion without more context?

Well, what that statement means is you want to write the best software you can with the lowest cost per feature. Easy to say. But how do you do that? Part of that is understanding business users and their language. Then you set about mapping some immediate business needs into the technology domain. That means understanding developers and their language.

That’s two different languages or abstractions.

Testing, as an activity, can provide a glue between those two languages, allowing both to be as expressive in their own domain as necessary but distilled into a common abstraction to allow making informed decisions about correctness and value.

Correctness and value are … you guessed it … “things that matter.”

Okay, let’s consider another statements about metrics:

• One of the few code-quality metrics that has any validity: how easy it is to find incorrect behavior and add new behavior.

Maintainability over the life of the code is (usually) more important than optimizing its present state. So you (ideally) write applications that can succeed in the immediate present and still adapt to the future. It’s a truism that the cost of change will eventually eclipse the original cost of the application. That means understanding a key point, which is that design is navigating the decision points that have consequences for the future.

Thus design is another of those “things that matter.”

(See how that original title keeps coming up? There’s a reason for that!)

A Focus on Design

What this thought about design leads us to is the following:

• Good design communicates its intent.
• Good design is about making decisions around choices.

Those decisions often focus on what we think is likely to change in the future versus what is likely to remain more or less the same. So here thinking about metrics has allowed us to think about some other “things that matter:” design and its relationship to the likelihood of change. But we can go a step further:

• The purpose of design is ultimately to reduce the cost of change.

Testing is, in large part, how you put pressure on design such that design is kept as cheap as possible at all points. In fact, testing has to put pressure on design at a pace that matches how product teams specify and how development teams write code. There are two paths here:

• From customer conversation down to where it intersects with code.
• From code conversation up to where it intersects with the user.

Keeping the cost of change low means being able to be flexible along both of those paths.

How Does This Tie Into The Core Metrics?

Okay, so if you keep in mind this focus we ended up with, you probably can see how all of what I just said factors into those core metrics I talked about earlier.

Feature coverage is about how much we understand about our features such that we believe we can test them to determine if they provide adequate value for their intended users. Feature readiness falls right out of that by allowing us to say what is “ready” based on both objective and subjective notions of quality around the value being provided. That, in turn, allows us to focus on how ready we are for those features to make their way to production, where we can get even more feedback about the value we are providing.

Delivery Team Approaches

Think of how the above intersects with various approaches.

• Agile focuses on early delivery; it handles changing values.
• Lean focuses on cheap delivery; it minimizes waste.

Easy enough, right? Now this:

• The theory of options suggests you shouldn’t exercise your options before you have to. That’s agile.
• The theory of constraints is all about making systems efficient and avoiding bottlenecks. That’s lean.

Notice how the fact that we “practice agile” or “practice lean” is much less important than exactly what we do. And what we do ultimately comes down to a lot of design. And that design, ultimately, gets enshrined in code that provides business features that are said to enable value. I’ll trust I don’t have to spell out how relevant this is for product development and product management and I’ll also hope that the intersection with testing is pretty clear.

This is why acceptance testing, in particular, matters because ultimately it is going to be the barometer with which we are able to determine if we are providing value.

So, again, in my previous post, I mentioned two things that fall out of product acceptance. One was a focus on things that matter. Here, specifically, I talked about metrics that matter and how — when we keep to just a few very operational metrics — we find other things that matter. I also mentioned that testable requirements fall out of this approach. I’ll cover that in the next post in this series.