The question of this blog title comes up often. The worst answer that can be given is: “When there are no more bugs.” It’s the worst answer because the inevitable follow up is: “But how do you know?” On the other hand, some people, upon answering this, begin providing a very convoluted answer. Here’s my take.
I honestly don’t know that I ever stop testing, per se. After all, testing extends back into the specification process before implementation even occurs. Testing also extends forward into deployment and involves the application living in production, including such aspects as coordinating with support.
But that can sound like I’m avoiding the question. So if reframe this a bit, I can ask not when I stop, but rather: When do I get to the point of saying “I’ve tested enough for this particular feature”? I would say that moment occurs when there is a preponderance of evidence that strongly suggests a low likelihood for finding any more impactful enough bugs.
Here “impactful enough” means bugs that would severely diminish or entirely reduce the value experience people get from the product or service. That’s all, in part, a value judgment and that judgment may change based on the type of feature or the context of the feature. Maybe it’s going to be part of a demo at a conference. Maybe it’s being provided to some early adopters. Maybe it’s part of a promised delivery to a third party.
One of my concerns, though, is that when this “when do I step testing” question is asked it also seems to be implicitly framed as treating testing solely as an execution activity. If you also treat testing as a design activity, then I would argue that you never stop.
Well, unless your company has stopped designing features entirely, which is unlikely.
I think it’s important for testers to keep in mind that testing is a broad-spectrum, continuous activity that — for specialist testers, anyway — never truly stops. As an activity, it simply undergoes shifts of focus and emphasis based on context.
Great post. Thanks for the inspiration!
I’ve found that for me the easiest answer is this: ”When we have invested all of the time and money planned for this” 🙂
As you say you are never really done. For me the questions often becomes if my time is better spent doing something else.
Of course that question is not trivial to answer. Low likelihood of finding impactful bugs in a feature definitely plays a big role. Opportunity cost is another part of it.
But if opportunity cost makes you stop testing even though there is still a relevant likelihood of finding any impactful bugs, then this of course has to be made clear to stakeholders.
It is definitely an interesting question.
Agreed. And it’s important to keep in mind that we’re not just testing what’s in front of us right now (in terms of functionality delivered). I also have an eye on internal qualities, often working with the developers for that. I’m also looking at how upcoming features are being specified and understood, ideally putting in my test design then so my test execution, later, is a bit easier.
I feel this article’s title is a bit of a “traditional question” — sometimes even used as a gauge during interviews — but it how that tradition is adhered to (how the question is answered) actually gets flipped a little bit if testing is reframed as a wider-lens activity.
Our decision that it’s OK to stop testing is based on a feeling; not a fact. But acting on that feeling is okay if we have thought sufficiently critically about the situation. We can’t verify today that we would not find an impactful bug tomorrow. That can’t be known until tomorrow comes around.
We’ve said for a while that testing never ends; it only stops. I think you DO stop testing, Jeff. I mean, you stopped long enough to write this article.
Every means to decide when to stop testing is a heuristic, and there are bunch of them. But they all add up to the principle that you’ve laid out very well in this post: we stop testing when the value of continuing is lower than the costs.
http://www.developsense.com/blog/2009/09/when-do-we-stop-test/
Cheers,
—Michael B.
When you say “our” — who are you speaking for? Not for me, surely. And it takes more than one to make an “our.”
In any case, I tend not to follow the Tyranny of the Or. I believe you could “stop testing” based on both feeling and fact. As far as do we EVER stop testing, I think it’s important to recognize context. Obviously the context I’m talking about is our professional context. Do I stop testing when I leave work and walk to my train? When I’m on the train? Whether I do or not is entirely beside the point. So did I stop testing when writing this article, as you assert. Perhaps. Or perhaps, at least in this case, I was continuously testing my ability to articulate my own points.
In my experience, you stop testing when something more urgent or with a tighter release date lands on your desk.