Years ago I asked about what makes testing complicated. At that time I didn’t really have a very distinct nuance between “complex” and “complicated.” But I think my instinct was accurate. So here I want to focus on what makes testing complex (which is often inevitable) and that can help frame what makes testing complicated (which is not inevitable).

I want to look at this question in a way that is likely dissimilar to the way many testers would answer it. That doesn’t mean I’m right or even have a good point over other testers. I simply don’t want to repeat what you’ve probably read a thousand times before.

First I’ll start with what I think is an interesting quote:

“On two occasions I have been asked, ‘Pray,

Mr. Babbage, if you put into the machine wrong

figures, will the right answers come out?’ . . . I am

not able rightly to apprehend the kind of confusion

of ideas that could provoke such a question.”

— Charles Babbage, Passages from the Life of a Philosopher (1864)

I feel sort of the same way as Mr. Babbage, in terms of the inability “rightly to apprehend,” when I hear something like this from testers:

“Without requirements or design, programming is the art of adding bugs to an empty text file.”

I realize this is an attempt to be clever but there are a disturbing number of people, including and especially testers, who believe the first part of that statement can in any way be true.

There is rarely ever truly a case of “without requirements or design.” There are many cases of only implicit requirements, as opposed to explicit ones. But requirements pretty much always exist. And, to be sure, there are cases of unreflective (and thus expensive) design. But design pretty much always exists.

And, of course, it’s quite possible to have requirements and design and still add bugs to those empty text files. This is often because we can have design that isn’t kept as cheap as possible at all times; we can have requirements that have subtle and overt opacity to them.

All of this is no mere semantic nitpick; it’s actually fundamental to understanding how technical debt actually occurs and how we go from complexity (which is okay) to complication (which is not). And thus it’s fundamental to understanding not just the shifting nature of quality but pinning down the ways in which quality degrades.

This is the field upon which the battles of testing are waged. When these battles are not waged, this is a good way that our projects (and thus our testing of them) shifts from complex to complicated.

But, more than that, the above offhand comment suggests something even more pernicious: that it’s possible to have a project without history. Requirements and design are a reflection of history.

And, as I’ve said before, testing is like studying the past. This is very much another aspect of what makes testing complex. In fact, I would argue it’s one of the core aspects of the complexity of testing. It’s a challenge that can be extremely difficult to articulate in modern companies however who are often operating under an “execution bias” and don’t see what I’m describing here as a form of execution.

The Traces of Our History

Okay, let’s start breaking this down a bit.

Many of us are working in a technology context of some sort. This means computers are part of the picture. Many of the decisions we take as part of a delivery team to bring features out to the world are simply told to the computer to execute. This is done via a form of communication called “coding.” This communication is encoded in a form of documentation called “source code.” This encoding is done using a formal language called a “programming language.”

Yes, I know, I seem like I’m belaboring a terribly obvious point.

But make no mistake: what we are doing is passing knowledge and decisions to the computer in a form it can understand. We are essentially taking fluid history and making it concrete by encoding it in a static form. (And keep in mind the project singularity that is generated as history occurs.)

Having the computer understand the source code is not the hard part, however. The hardest part is for people to understand what has been done, in order to do better and faster work. But to also maintain a level of quality while being “better” and “faster.”

Again, I can’t stress enough what I believe is a key point: software development is all about knowledge and the decision-making based on it.

There’s not just the stated problem and what we did. There’s also the reason why we did what we did and the facts and evidence that led us to that approach; and perhaps even the alternatives we considered and discarded.

It’s critical for us to be able to compare a mental view of the intended system with a generated view of the actual system. This can help spot inconsistencies, ambiguities, and outright contradictions.

And that’s part of what testing does. Or, at least, what it should be doing. And that’s part of what makes it complex. Testing, in part, helps uncover the history that led to what is. Testing is not just a reflection of what is.

I’ll sort of expand on something I said in my About Testing post here. Epistemology is about the way we know things. Ontology is about what things are. Ontogeny is about the history of changes that preserve the integrity of something. Thus, as a specialist tester, I reduce the epistemological opaqueness and ontological confusion due to cognitive biases we have when we perform ontogenetic change across the boundaries where humans and technology intersect.

Yes, that is quite the mouthful, now that you mention it. But I fundamentally believe that when you distill all testing down beyond its pure mechanisms, the above is exactly what it is. And it is a challenge for modern testers to practice that in an industry that routinely doesn’t want to hear it but would rather turn testing into a programming problem.

That being said, let’s not lose our thread here. What does the above tell us? It tells us that code and tests are a consequence of applying knowledge into a format that can be executed.

In that context, all the time we spend on finding the missing or incomplete knowledge is time not spent on making better decisions about how to provide experiences that add value. And making better decisions (particularly sooner) that are focused on value is what testing can help people do. It’s part of what makes testing complex.

When testing does not act in this capacity, that is part of how our projects (and thus the testing of them) shifts from complex to complicated. It’s how we get a disconnect between the business domain and the technology domain that supports it.

The (Subtle) Forces on Our History

Okay, but hold on. We started off saying that ultimately we will have code. I don’t want to lose sight of this seemingly obvious point.

Consider that complexity is largely inevitable and not a bad thing. Complication, on the other hand, is almost always a bad thing and is not inevitable. A natural law of the universe is that as any coordinated system grows, the complexity needed to keep it glued together grows even faster. Beyond that, information theory teaches us that compressed data is statistically similar to random noise. Finally, there seems to be a “law of conservation of complexity.” This basically says that we can’t avoid complexity, we can only redistribute it; we can push it around or attempt to distribute it.

Consider a salient, if poetically stated, point:

“Below the surface of the machine, the program

moves. Without effort, it expands and contracts.

In great harmony, electrons scatter and regroup.

The forms on the monitor are but ripples on the

water. The essence stays invisibly below.”

— Master Yuan-Ma, The Book of Programming

I’ve talked about project forces in the context of our abstraction stacks and related to our testing. I think what I’m getting at here in this article is a specific project force regarding how projects evolve and how to look out for the pathology for when (necessary) complexity starts to become (unnecessary) complication.

Looking for this force, I think it’s important to note that “complexity” has little to do with the size of the system in question. The complexity comes from how many variables are interacting within that system. And once you introduce humans into a business context, you end up with a lot of variables. It’s not just the number of humans but how they interact with each other, and the decisions and micro-decisions made that take us from inputs to outputs, and the means by which those decisions are recorded and understood (requirements) and conceived of as adding value (design).

Systems with small numbers of variables lend themselves to modeling. Systems with many variables generally don’t. The only way you can explain the behavior of those many-variable systems is to simulate them. Simulating them sounds a bit like testing them, right? But also note that to simulate them means you trace the (or a) history. And, as I’ve said, testing is a means of providing those traces of history.

The Mechanisms of Our History

Consider approaches like Lean, Agile, Scrappy, and so on. We generally tout these as ways to grow our efficiency and productivity. Yet what’s interesting is that we think our growth in efficiency and productivity presumes a continued ability to understand all of this engineering that we construct as part of our projects.

Along with that, we feel that if we have the ability to scrutinize our systems, we can break them down, such as via code and test coverage. And if we can do that, we must be able to determine how those systems work. And if we can do that, we must be able to determine how they can fail. And thus, we presume, do we understand these systems.

And that makes a certain amount of sense, right? Testing, after all, is largely predicated on the idea of determining when something fails and finding the conditions under which that failure can occur.

The challenge came as our abstraction stacks got larger. Strategies like the creation of hierarchies and the use of abstraction have, in the past, helped us manage complexity. The problem we have now, however, is that we tend to have way too many abstractions. In this context our bits of technology — our code and our tests — intersect and interact across those many levels of abstraction. These various intersections are often leading to interactions that we are unaware of and sometimes can’t even imagine. And even if we can imagine them, simulating them can become it’s own sort of nightmare.

What we end up with is an abstraction stack that is made up of a web of relations that produces what seems to be emergent behavior. We get unexpected outcomes under various conditions, only some of which we can simulate or simulate accurately enough for reproduction. Providing yet an extra challenge to this, we even end up with a higher-order type of interconnection that’s generally referred to as interoperability. In other words, whatever we are constructing has to operate between systems or technologies, many of which we have no control over. And those technologies have the same issues that our own technologies have!

I trust it’s fairly obvious that this is a key point regarding why testing is complex.

Not only can it be hard to recover our history but it can be hard to even determine when something might fail. And, in fact, those two points aren’t as disparate as they seem. After all, a failure path is a specific path through a history. And imagining failure paths (requirements) and providing a means to handle them (design) is a key part of realizing a particular history that we believe adds value.

Testing to Manage History

Bringing a certain point home here: I believe one of the core aspects that makes testing complex is that testing, in part, enables a cumulative process of knowledge management. Knowledge doesn’t exist in isolation, of course. Knowledge needs to augment with the extra context, intent, and rationale that exists as part of our projects. Expressing the knowledge literally into code or tests is effectively what approaches like Clean Code, DDD, TDD, BDD, ATDD and others are doing.

Or at least claim they are doing.

In fact, they’re not really doing any such thing unless we humans utilize them with that goal in mind. Putting it simply, we want to bridge the language of the code (implementation) with the language of the business domain (intent). Arguably, that’s a huge part of what testing is about. And it’s a large reason why it’s complex.

We create knowledge as part of our projects. We do this with data that we use to derive information that we consider to be knowledge. And that knowledge allows us to provide business value. But to do that we have to be able to ask (and, ideally, answer!) some questions:

• How does that knowledge become part of our collective wisdom?
• How does that knowledge become something we can reason about in the future?
• How does that knowledge become something by which we can shine a light on decisions made, both in terms of rationale and effect?

I’ll close this with a great quote from Dylan Marron who runs the Conversations with People Who Hate Me podcasts:

“I want us to move toward a system where we are operating on at least a shared pool of truth rather than separate truths we can all link to.”

Oddly enough, given the source, I think that is a perfectly good way of summing up what testing, as a wide-angle-lens specialized discipline, should be striving for and why it is such a complex discipline.