Thankfully most testers that I come across do realize that the notion of having “zero defects” is, in fact, a fallacy. But this notion of something being “defect free” still persists in the wider industry. And it’s important to quash that perception. How I frame this when encountering the thought differs a bit. So here I’ll give a brief overview of various ways I respond.
The Zero Defect Philosophical Problem
Philosophically speaking, the very idea of zero defects is a statement of total knowledge leading to perfect predictability.
When put that way, it sounds like a ridiculous position to hold, right? People who hold this view are akin to scientists before, say, the mid-nineteenth century who had a malformed view of science which suggested that science aimed for completion and perfection.
The Zero Defect Framing Problem
When testers hear “defect” they often equate it with “bug.” Usually this is relating to a “bug” in a software context. And that can be entirely fine. But those definitions can actually differ depending on the industry you are in. For example, you might be in an industry that does make distinctions between “error”, “defect” and “failure.”
But there’s a way to bridge this and I’ve found it to be applicable pretty much regardless of industry.
Let’s frame “defect” the way we frame “bug.” By which I mean most modern testers say that a “bug” is anything that threatens value. Some threats to value will be objective: this application crashes; this blender catches fire; this autonomous vehicle can’t distinguish road stripes. Others will be subjective: this is ugly; this is confusing; this is too slow.
As such, with this framing, you can never have “zero defects” because there can always be a perceived threat to value.
Quality can be a shifting perception of value over time and, as such, how would you ever claim to have “zero defects”?
The Zero Defect Evidence Problem
Even if there were truly zero defects in something, you could never prove it.
For the objective value threats, you can perhaps reduce their incidences such that they rarely happen. Perhaps a given incidence is so rare that, for all practical purposes, it never occurs. Thus that particular defect may have been eliminated.
Which says nothing, of course, about other possible defects.
And, again, that’s only treating defects as objective aspects.
Further, it’s quite possible that the objective aspect for me doesn’t apply for others. Stuck brakes, for example, were a problem with some vehicles and not others and, even with the problematic vehicles, only occurred only under certain conditions. Would I be fair to say there were “zero defects” (of that kind) since it never happened to me? Or to anyone I personally knew? No, because it clearly happened to some, whether I was aware of that or not.
And, of course, how do you achieve “zero defects” when a defect can be subjective as I indicated above? What’s a great feature to one person can be terrible to another. What a lot of people like, a lot of other people may dislike.
It really is as simple as asking whether you can realistically prove a “zero defect” claim.
If you can’t prove it, then why talk about it? What reality does it have? And if it has no reality, why would you make the claim for it?
Again, you can talk about a specific type of defect and say you don’t observe that particular one. In fact, perhaps nobody observes that particular defect. At least under any realistic conditions. But even that doesn’t say there are “zero defects.” It says there is an absence of the one that you happen to be focusing on.
So, ultimately, a statement about “zero defects” is telling you something about the standards of evidence someone holds and what someone is willing to take as proof.
The Zero Defect Claim Problem
If you make a claim of “zero defects” and you are providing something as a product or service, you may be making a claim that could be challenged legally. You yourself may not have to deal with this but your company or those you represent may now have to be called upon to show that the claim of “zero defects” had any reasonable chance of being true.
And, as I hope I’m indicating, there never was a reasonable chance of the claim being true.
Thus this can be a dangerous claim to make (personally and professionally) and, given what I said above, I think it’s a misleading and irresponsible claim.
Going back to the philosophical side and aligning that with the legal side, you simply can’t prove zero defects. You can’t prove a negative. You can try to prove a certain level of defect tolerance — which for all practical purposes may be zero — for particular types of defects. This is what formal verification does in various industries, for example. This can be then be used as the basis of justifiably demonstrable claims of the best effort being put forward to search for and mitigate defects.
And, by the way, all the formal verification procedures in the world don’t matter if they can still let certain classes of problem through. We’ve seen this most recently with issues around airplanes, which do have formal verification processes in place.
This disconnect between a usually very thorough formal verification procedure and “hidden” defects most often happens when there are various aspects of interrelated design, sometimes between hardware and software.
This also happens when there are components of your system that are entirely “black box” in nature. We’ve seen incidences of the latter most recently with machine learning and artificial intelligence systems where a massive amount of testing is done on the systems, but where the algorithms themselves are still essentially unknown in terms of what exactly they are going to do given certain conditions. The most pernicious examples are those where bias is tested for very thoroughly and then the system is released and immediately displays a great deal of bias.
The Zero Defect Ideal Goal Problem
For some people this notion of arguing against “zero defects” is taken as almost a personal assault on reason. They see this as either a gross form of apathy or negativism. Such people may say that even if “zero defects” aren’t possible, we should still strive for that goal.
But should we?
Because then the question is asking if we should strive for an impossible goal with an impractical method.
Make no mistake about it: that’s really what we’re saying. We’re talking about removing all uncertainty under all conditions such that we can assert a given system works exactly — and only! — as we say it does. Which also entails removing any ambiguities or inconsistencies — defects! — in our own language and thus how we describe our system, such that there is no possibility at all for misuse or confusion among the user base.
That’s an ideal state if we can get to it. But human history argues we likely can’t, particularly that part of human history wherein humans design and build complex things.
What we can strive for is to reduce those defects that are the most impactful, certainly in terms of adverse consequences, whether that be to life itself or to aspects of life, such as monetary or other considerations. We can do the same for that which is impactful to frustrations with use of products, even if not life-threatening.
Those are possible goals with established practical methods.
Should those goals and methods somehow lead us to a world of literally zero defects — no adverse consequences, perceived or actual, under any conditions whatsoever for any person in any situation — well, all the better.
The Test Mandate Is Clear
So, in the meantime, let’s deal with reality. And the reality is: there will always be defects. Even if there are no defects, you won’t be able to prove it. So you may as well act as if there are defects.
Does that sound like a sad reality to live in? Maybe. Maybe not.
Either way, that’s why the industry needs test specialists.
We not only help dispel illusions and delusions (like “zero defects”) but we provide techniques and approaches to deal with the consequences of reality (which is that defects exist), the limitations of cognition (which allows defects to be created), and the failures of intuition (which allows defects to persist).