The Zero Defect Fallacy

Thankfully most testers that I come across do realize that the notion of having “zero defects” is, in fact, a fallacy. But this notion of something being “defect free” still persists in the wider industry. And it’s important to quash that perception. How I frame this when encountering the thought differs a bit. So here I’ll give a brief overview of various ways I respond.

The Zero Defect Philosophical Problem

Philosophically speaking, the very idea of zero defects is a statement of total knowledge leading to perfect predictability.

When put that way, it sounds like a ridiculous position to hold, right? People who hold this view are akin to scientists before, say, the mid-nineteenth century who had a malformed view of science which suggested that science aimed for completion and perfection.

The Zero Defect Framing Problem

When testers hear “defect” they often equate it with “bug.” Usually this is relating to a “bug” in a software context. And that can be entirely fine. But those definitions can actually differ depending on the industry you are in. For example, you might be in an industry that does make distinctions between “error”, “defect” and “failure.”

But there’s a way to bridge this and I’ve found it to be applicable pretty much regardless of industry.

Let’s frame “defect” the way we frame “bug.” By which I mean most modern testers say that a “bug” is anything that threatens value. Some threats to value will be objective: this application crashes; this blender catches fire; this autonomous vehicle can’t distinguish road stripes. Others will be subjective: this is ugly; this is confusing; this is too slow.

As such, with this framing, you can never have “zero defects” because there can always be a perceived threat to value.

Quality can be a shifting perception of value over time and, as such, how would you ever claim to have “zero defects”?

The Zero Defect Evidence Problem

Even if there were truly zero defects in something, you could never prove it.

For the objective value threats, you can perhaps reduce their incidences such that they rarely happen. Perhaps a given incidence is so rare that, for all practical purposes, it never occurs. Thus that particular defect may have been eliminated.

Which says nothing, of course, about other possible defects.

And, again, that’s only treating defects as objective aspects.

Further, it’s quite possible that the objective aspect for me doesn’t apply for others. Stuck brakes, for example, were a problem with some vehicles and not others and, even with the problematic vehicles, only occurred only under certain conditions. Would I be fair to say there were “zero defects” (of that kind) since it never happened to me? Or to anyone I personally knew? No, because it clearly happened to some, whether I was aware of that or not.

And, of course, how do you achieve “zero defects” when a defect can be subjective as I indicated above? What’s a great feature to one person can be terrible to another. What a lot of people like, a lot of other people may dislike.

It really is as simple as asking whether you can realistically prove a “zero defect” claim.

If you can’t prove it, then why talk about it? What reality does it have? And if it has no reality, why would you make the claim for it?

Again, you can talk about a specific type of defect and say you don’t observe that particular one. In fact, perhaps nobody observes that particular defect. At least under any realistic conditions. But even that doesn’t say there are “zero defects.” It says there is an absence of the one that you happen to be focusing on.

So, ultimately, a statement about “zero defects” is telling you something about the standards of evidence someone holds and what someone is willing to take as proof.

The Zero Defect Claim Problem

If you make a claim of “zero defects” and you are providing something as a product or service, you may be making a claim that could be challenged legally. You yourself may not have to deal with this but your company or those you represent may now have to be called upon to show that the claim of “zero defects” had any reasonable chance of being true.

And, as I hope I’m indicating, there never was a reasonable chance of the claim being true.

Thus this can be a dangerous claim to make (personally and professionally) and, given what I said above, I think it’s a misleading and irresponsible claim.

Going back to the philosophical side and aligning that with the legal side, you simply can’t prove zero defects. You can’t prove a negative. You can try to prove a certain level of defect tolerance — which for all practical purposes may be zero — for particular types of defects. This is what formal verification does in various industries, for example. This can be then be used as the basis of justifiably demonstrable claims of the best effort being put forward to search for and mitigate defects.

And, by the way, all the formal verification procedures in the world don’t matter if they can still let certain classes of problem through. We’ve seen this most recently with issues around airplanes, which do have formal verification processes in place.

This disconnect between a usually very thorough formal verification procedure and “hidden” defects most often happens when there are various aspects of interrelated design, sometimes between hardware and software.

This also happens when there are components of your system that are entirely “black box” in nature. We’ve seen incidences of the latter most recently with machine learning and artificial intelligence systems where a massive amount of testing is done on the systems, but where the algorithms themselves are still essentially unknown in terms of what exactly they are going to do given certain conditions. The most pernicious examples are those where bias is tested for very thoroughly and then the system is released and immediately displays a great deal of bias.

The Zero Defect Ideal Goal Problem

For some people this notion of arguing against “zero defects” is taken as almost a personal assault on reason. They see this as either a gross form of apathy or negativism. Such people may say that even if “zero defects” aren’t possible, we should still strive for that goal.

But should we?

Because then the question is asking if we should strive for an impossible goal with an impractical method.

Make no mistake about it: that’s really what we’re saying. We’re talking about removing all uncertainty under all conditions such that we can assert a given system works exactly — and only! — as we say it does. Which also entails removing any ambiguities or inconsistencies — defects! — in our own language and thus how we describe our system, such that there is no possibility at all for misuse or confusion among the user base.

That’s an ideal state if we can get to it. But human history argues we likely can’t, particularly that part of human history wherein humans design and build complex things.

What we can strive for is to reduce those defects that are the most impactful, certainly in terms of adverse consequences, whether that be to life itself or to aspects of life, such as monetary or other considerations. We can do the same for that which is impactful to frustrations with use of products, even if not life-threatening.

Those are possible goals with established practical methods.

Should those goals and methods somehow lead us to a world of literally zero defects — no adverse consequences, perceived or actual, under any conditions whatsoever for any person in any situation — well, all the better.

The Test Mandate Is Clear

So, in the meantime, let’s deal with reality. And the reality is: there will always be defects. Even if there are no defects, you won’t be able to prove it. So you may as well act as if there are defects.

Does that sound like a sad reality to live in? Maybe. Maybe not.

Either way, that’s why the industry needs test specialists.

We not only help dispel illusions and delusions (like “zero defects”) but we provide techniques and approaches to deal with the consequences of reality (which is that defects exist), the limitations of cognition (which allows defects to be created), and the failures of intuition (which allows defects to persist).


About Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.
This entry was posted in Testing. Bookmark the permalink.

One Response to The Zero Defect Fallacy

  1. Robert Day says:

    I think you’re preaching to the converted here. The issue isn’t about how many defects, or even how impactful defects are. The real, everyday issue is how to persuade influencers that we will do our best but the world really isn’t made that way. But in their world, there are absolute answers and things that really shouldn’t happen; and when stuff goes bad in a big way – such as in aviation software systems, as you said – it’s hard to get acceptance of realities. Add in to that evidence of corporate corner cutting (again, as in the fallout from the aviation industry example we are all thinking of), and our argument that “zero defects are not achievable” does not get a hearing, even if everyone is being reasonable.

    Let me give you an example.

    In a previous life, I worked for Ofwat, the UK water industry economic regulator. One of the big issues in the water industry is leakage. Customers complain when water pipes leak, especially when there are summer water shortages. “Repair all the leaks!” is the cry. But the costs of repairing a leak are pretty much the same whether it’s a leak in a half-inch supply pipe to a single property or a six-inch strategic main supplying half a city. So Ofwat developed something called the “economic level of leakage”, a methodology that determined the point at which it is cheaper to bring new water sources into supply to make up for the leakage than it is to track down and fix the smaller leaks.

    We commissioned some focus groups on this, and they all found the same thing. We asked the groups if they agreed that all leaks should be fixed, and they all said yes. Then we explained to them, in the simplest possible terms so as not to leave any doubt, about our economic level of leakage measure, how it worked and gave them simple, worked examples so that they could understand it. We even asked if they understood it, and everyone in the groups said that they did. So the new asked the main question again: do you think that all leaks should be fixed, irrespective of cost?

    The answer was still yes.

    I think the same would happen if you substituted “zero defects” for “zero leakage”. Influencers and stakeholders would still say that they expect zero defects even if you explain to them, slowly and clearly, that that is not possible. If you can think of a way around that, I think a lot of people would be very interested!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.