I find that many testers still like to “talk in quotes.” Meaning, they like to throw out quotes or sentences and then act is if they’ve said something profound. And maybe they have. But I’m seeing more of this lately without, it often seems, the necessary ability to think beyond the quote. Let’s dig in and see if I have a point to make.

I should note that here I’m treating “culture” in the sense of patterns of behavior that aren’t dictated by some procedure. These patterns of behavior are often noticed by others. And, as such, those patterns can be used to judge the basis of a craft or discipline. At the very least, it can be used to judge the practitioners of that craft or discipline.

I don’t know if my overall point here goes to the dogma and tradition I’ve talked about before but it certainly intersects with my discussions around received wisdom.

Rather than rehash some of that, I’ll instead ask you to consider the show The Amazing World of Gumball. Basically you have the main character Gumball Watterson, a 12-year old that tends to get into trouble, and his sister Anais. Anais is the youngest and most intelligent member of the Watterson family despite being only four years old. One bit of her wisdom was relevant for my post:

I want to mention Damian Synadinos and his post here for the above image. I’ll also mention a related point which was the show Adventure Time (with Finn & Jake). This was a show about a boy (Finn) and his shape-shifting dog (Jake), both of whom often showed unique wisdom. One of the episodes had this situation:

And the quote associated with that scene was:

“It’s hard to step outside yourself when you are enmeshed with another being.”

The idea being, of course, that quoting someone — using their ideas rather than your own too much was — essentially become “enmeshed” with them, thus harder to set outside of yourself.

It’s not that utilizing quotes is bad, of course. Rather, it’s when the utilization of quotes is presented in place of substance. When the utilization of quotes is used as a crutch (even if unconsciously) to do any actual thinking or, at least, thinking beyond the quote.

Maximizing Quality

I’ll spend a little more time on this first type of quote just so you can get a feel for my thinking. This one, or similar, is a type that testers like to trot out:

“Testing helps to maximize quality.”

When testers toss this one out there, I have to ask: Do they really believe this? I say that because I know that to “maximize” means to make as large or great as possible.”

But “as large” and “as great” are vague requirements, right? If someone gave the development team a requirement that we should build a feature “as large and as great as possible,” what would that mean?

Two other points are perhaps obvious but worth mentioning. Quality is subjective. The same people can look at something and see varying levels of quality. Even the very same person can have different perceptions of the quality of the same thing based on their mood. So definitionally you can’t maximize something like that.

The other point is that “quality” is a vague term itself until you realize we talk about internal and external qualities. Which of those do we maximize? Some are in a tug of war with others, which means as we bring one up, we might cause another to go down. Modern testers, I tend to find, are notoriously bad at these distinctions, assuming they even think of them at all.

Key to this is the idea of maximizing, though, and thinking about that word. Whether someone who is throwing this quote out believes that testing “helps” to maximize quality or that testing actually does maximize quality is irrelevant if “maximizing quality” is not possible. And whether maximizing quality is possible or not is irrelevant if we can’t prove it. And we can’t.

Good testers are focused on what can be proved and what cannot. So unless someone is using this quote as an example of how not to think, this tends to show a compromised view of testing. That means a testers who is genuinely throwing this quote out there probably has a compromised view of testing. And that’s a tester that I don’t want representing my craft and discipline.

So what do we say instead? Testing is one of the components by which we can help people make decisions about quality. Also, quality is subjective as well as objective. That means you can’t possibly “maximize” quality in some categorical sense. Quality is a shifting perception of value over time. What testing can do is point to degradations of those things that we have collectively agreed are quality. That’s a lot easier to prove than to try to prove we somehow “maximized” quality. So if someone did want to quote some nuggets, I would suggest some of those that I just mentioned.

You’re Doing It Wrong

Here’s another one that was made the rounds lately:

“If testing is holding you back, then you’re probably doing it wrong.”

This is one of those quotes that I assume is meant to sound clever in an immediately concise way.

But consider: what does “holding you back” mean? Introducing a delay — one way of “holding back” — can be a good thing. Otherwise, why don’t we just develop in production? So testing does and should hold us back from certain things, such as simply deploying whenever we want to without some sort of filter to determine if what we’re deploying is wise.

Thus if testing is “holding you back” — you’re probably doing it exactly right! The question is whether testing is holding you back excessively or without contributing to any significant understanding about your quality (or lack thereof).

Testers who trot out quotes like the above rarely, in my experience, make these distinctions. And, again, that’s a tester I don’t want representing my craft or discipline.

I would also add that “holding back” doesn’t have to mean “not moving forward.” That frames it as an either-or dichotomy, something testers seem to enjoy doing. I can be holding back on deploys while moving forward on gathering information on risk. For me this is a spectrum, not a binary state.

So what can we say instead? Testing does many things, one of which is put pressure on design. That can minimize delivery time by making sure that testability is one of the primary quality attributes, which leads to other things, such as making it easier to add and change behavior, reducing the technical debt we saddle ourselves with and so on. Sometimes testing does that while holding certain things back a bit while moving other things forward a bit.

Testing is often about finding that balance. If you want to quote something, figure out a way to put that in a concise quote.

Anything About Checking

I’ve talked a lot about the checking distinction, which I’ve called a flawed argument but have revisited the idea to see if I still feel the same way. (I do.)

I bring this up because there are many examples out there of the “checking quote” variety. Here’s one that I find particularly poor:

“Checking turns software testers into robots, exploration turns them into investigators.”

This is why “checking” is a bad way to frame this. I won’t revisit that argument here but, no, checking doesn’t turn someone into robots necessarily. Ask experimentalists in many disciplines if their periodic checks make them robots and they’ll likely tell you that you have no understanding of the scientific method.

Likewise, using this terminology, exploration can also turn someone into a “robot” (i.e, unobservant, unreflective). Exploring without the connotation of experimentation and investigation means nothing in and of itself. The above quote might suggest that exploration makes you an investigator. But it’s not automatic. And, again, all you have to do is ask people who work in the sciences.

So what might we say instead? Testing, like science, is basically an attempt to understand how, why, and under what conditions things happen the way they do. Any such attempt at understanding shows that a basic underlying methodology exists and emerges, some of which involves rote actions (which is a type of testing) and some of which involves exploration (also a type of testing).

Once again, it can be an “and” rather than an “either-or” which is a common pattern of those quotes from testers: “either you are or aren’t doing it wrong; either you are testing or you are checking.”

Simplified Quotes

Sometimes testers just quote other well-known testers or quote something and attach it to well-known testers, even if the quote isn’t quite accurate. As an example, the following was making the rounds on LinkedIn and was attributed to Cem Kaner:

“Developers find and fix most of their own bugs, what testers find are what developers missed.”

It’s a very limiting view of testing, I think. First of all, what the above quote could suggest is that testers are this afterthought that might not even be needed if only developers were slightly better at their jobs.

Beyond that, this quote treats testing as (mainly, perhaps solely) an execution activity. But when framed as a design activity, developers don’t find and fix most of their own bugs — especially when you consider bugs that impact internal qualities, not just external ones. This is where testers — or, more appropriately, the discipline of testing — puts pressure on design.

That pressure on design isn’t even just at the code level. It can be at a higher abstraction, such as the architecture that supports the code. Or at the level of abstraction wherein we are talking about what to build in the first place and deciding the outcomes that most contribute to value.

Key to this is that distinction: it’s testing that can help find this. This doesn’t have to be done by someone with the defined role of “Tester” which, you’ll note, the quote is focusing on. Instead what we should be talking about is someone who is performing the act of testing, with all the techniques and thought patterns behind the discipline. That can be a developer. But more than likely, it’s the delivery team as a whole working under the idea of distributed quality and democratized testing.

Shifting Gears

But sometimes quotes are useful, right? They do allow us to concisely covey a salient point. So let’s consider that as well.

Testing and Learning

Here’s one that I think is quite accurate:

“Running out of test ideas means that you haven’t learned enough about your software.”

Very true. And a corollary to this. “… haven’t learned enough about your software OR the domain in which it is situated.” A lot of testers feel learning about “the software” is what is needed. And it is.

But so is learning about the domain it works within and the value it provides to users within that domain. This is how testers make sure to step outside the technology and think about how the technology intersects with people. It’s that intersection that leads us to think about finding problems with behavior rather than just finding confirmations of functionality.

Testing and Proving

Here’s another one that attempts to tell us what testing can or can’t do. This is quoted by testers a lot and is from Robert Martin:

“Testing shows the presence, not the absence, of bugs. In other words, a program can be proven incorrect by a test, but it cannot be proven correct. All that tests can do, after sufficient testing effort, is allow us to deem a program to be correct enough for our purposes.”

I present the quote that way although those of you who have read Martin’s Clean Architecture know that the quote, as shown above, misses that Martin here is quoting the first part (“Testing shows the presence, not the absence, of bugs”) from Edsger Dijkstra. The rest is Martin’s expansion of the idea as he sees it.

This one actually straddles the line for me. I say that because this isn’t entirely true. Testing can show the absence of bugs in certain areas of concern. If it couldn’t, testing would not be viable. What testing cannot do is show the absence of bugs in a universal sense.

This may seem like semantic nit-picking but it’s an important way for testers to qualify what we do. The notion of “proving a program correct” is not the same thing as “proving the absence of bugs” but this quote — or, rather, the combination of quotes — could make someone think that.

Even more importantly, testing cannot prove quality in a universal sense. For example, if we go by the idea that a bug is anything that threatens value, then the notion of what is and isn’t a bug may differ by user. (And may even differ by the same user under different circumstances.) Framing things this way helps testers think about value versus correctness, which in turn allows us to more critically question the notion of “proving absence locally” and “proving absence universally.”

Which, finally, allows us to consider the notion of qualities (value enhancers) and bugs (value degraders), which can put operational specificity around “good enough.”

The Quote Culture

So, with all this being said, what actually is my concern with this quote culture? My concern is that it’s often not clear to me that testers think beyond these quotes. That’s the descriptive part. I’ll come to a prescriptive part in a second.

Testing as a discipline is often under siege.

This is often, I have maintained, due to its own practitioners. As such, in my blog I do try to periodically highlight where our practitioners are perhaps damaging our own craft. A lot of times this is in the context of the idea space, where many ideas are spread and you can watch how testers react to the ideas.

A fertile ground for this is to connect with as many testers as possible on LinkedIn, watch what they post, and then watch how people respond. This is particularly the case in situations like the above, with the use of non-ironic quotes to convey what are presumably thought to be deep truths.

My problem is not so much that testers throw these quotes and sound bites out there but rather that they rarely seem to think beyond them. And if that’s a wider perception, testing as a discipline will continue to be under siege.

So the prescriptive part I mentioned above might be that testers, when doing the relatively mindless activity of quoting someone else, should — akin to Anais Watterson’s sentiment above — have to show that they do think for themselves as well. It could be that just by providing the quote, testers are signifying their agreement with it. But, as I hope I’ve shown, that’s often problematic with many of the quotes testers like to cast out there.

Science has a large component of questioning established wisdom, not just repeating it. Testing is a form of science. The conclusion, I trust, is inescapable.