I recently read the book called Uncertainty: Einstein, Heisenberg, Bohr, and the Struggle for the Soul of Science, written by David Lindley. It made me think a bit about the “soul of testing.”
I posted before about how I believe testers should be cross-discipline associative, so here’s an attempt at me practicing that a bit.
As the book Uncertainty makes clear, the kind of physics that Bohr and Heisenberg were promoting, with the idea of uncertainty at the center, was, with only a little melodrama, a battle for the soul of science. That notion of “soul” was predicated upon the human inability to comprehend the physical world. This is as opposed to aspects of the world simply being inaccessible.
A few years ago I talked about the “science and art” of testing. An important point for me is that science is the attempt to extract order from confusion while art is the attempt to make that order palatable for people of various skill levels and interest.
To be sure, in my mind, testing does exactly the same thing. While requirements specify how something should be and code specifies how something is, tests can do both — and serve as an executable means of specification as well. As such, testing works to extract order as well as to infuse order. This allows various stakeholders to participate in collaboration and communication. Just as science and art provide a shared means of looking at the world, but through different lenses, tests are a multi-faceted work product that does something quite similar.
Taking a few points from Lindley’s book, a few points drew associations for me. For example, consider this:
“Theoretical predictions are only as good as the assumptions behind them. In the endless back-and-forth between experiment and theory, it’s uncertainty that tells the scientist how to proceed. Experiments probe ever finer details. Theories undergo adjustment and revision. When scientists have resolved one level of disagreement, they move down to the next. Uncertainty, discrepancy, and inconsistency are the stock-in-trade of any lively scientific discipline.”
In a similar fashion, test execution is also as good as the test design that informs it. Testing is filled with theory about how to do testing more effectively and more efficiently. But there’s also the case that testing is subject to uncertainty: when a tester meets the thing they are to test, reality happens. Those interactions with the thing being tested — those experiments — force ideas about testing to undergo adjustment and revision. It’s when oddities are found that the job gets interesting and forces a more detailed look, which may refine our view of how to test.
Or how about this:
“Starting with Copernicus and Galileo, with Kepler and Newton, modern science evolved through the application of logical reasoning to verifiable facts and data. Theories, couched in the rigorous language of mathematics, were meant to be analytical and precise. They offered a system, a structure, a thorough accounting that would replace mystery and happenstance with reason and cause.”
This is largely what business analysts, developers, and testers do as they work together on projects. They evolve an understanding of the business domain and back up that understanding by proposed facts (test cases) and data (the data conditions applied to test cases). Both the communication and the act of testing provide a structure.
“The phenomena of nature might be inordinately complicated, but at bottom science must reveal order and predictability.”
Likewise, testing does the same. I’ve been told many times that a system that I’m testing is not exact. That we “can’t know” what to put for observable results. What this means then is that the system is non-deterministic. It means that we can’t expect an underlying order that should take us from defined input, through desired action, to expected output. Thus we do not have predictability. I’ve been told that on many projects. And I have never found this to be true. But it was up to my test design and test execution to ultimately prove that.
“If scientists of one generation, building on the work of the last, could see that they had yet to achieve their ideal, they could equally believe that those who came after them would finish the job. The power of reason implied the ineluctability of progress. Science would become more grandiose, more encompassing in scope, yet at the same time more detailed, more scrupulous. Nature was knowable — and if it was knowable then one day, necessarily, it would be known.”
In a similar fashion, when tests are treated as requirements and as documentation, this “encompassing scope” is largely what you are building up. Executable information that can be used from one “generation” to the next, whether that generation is measured in terms of iterations of the application or in terms of future team members after inevitable changes in staff. Yet we know, as physicists eventually figured out, that this job will not be “finished.” There is never really a “done.” But there will always be progress. And, yes, our test repositories will become larger; our individual test cases will accrete into scenarios and from those form into workflows. Those workflows will be descriptions of behavior. Yet, any any point, there is detail that is amenable to scrutiny. Our applications are always knowable — even if we didn’t know it yet.
“Scientists … pictured the natural world in its entirety as an intricate but inerrant machine.”
While our applications are ultimately deterministic and are perhaps intricate, they are most certainly not inerrant. Environmental factors and the human element will always contrive to allow for deviations.
“The trick was to define your science in terms of observations and phenomena that lent themselves to precise description — reducible to numbers, that is — and then to find mathematical laws that tied those numbers into an inescapable system.”
That worked for physics. It doesn’t work for testing. At least not the same way. But we can certainly make observations that lend themselves to precise descriptions. We tend to call those patterns, whether that be design patterns in code or test patterns in execution. Maybe we can call these micro-patterns. We find “laws” — further patterns of good practice; let’s call them macro-patterns — that allow us to utilize our micro-patterns within the context of a given system of implementation, deployment and operation.
“If ever scientists were daunted by their ambitions, it was because of the sheer complexity of the machine they were trying to tease apart. Perhaps the laws of nature would be too vast for their brains to fathom. Perhaps scientists would find they could write down the laws of nature only to discover they lacked the analytical and calculational firepower to work out the consequences.”
In the world of testing, this is where you end up with 10,000 or more test cases. Some barely manageable rat’s nest of test artifacts that manage to compete with the production code itself for complexity. This confusion extends back the very platform those tests are to be testing. But this also speaks to the practice of automation. While we can write down our tasks to the nth degree, being able to execute those tasks manually is often problematic. We can get answers. But it takes far too long for the consequences of those answers to be clear.
Whence the Soul?
So what is the soul of science? And how does that relate to the soul of testing?
Just as in science, it’s uncertainty that tells the tester how to proceed. A few of the attributes I want to see in testers is the ability to spot ambiguity, discover inconsistency, and ferret out discrepancies. Those attributes must be buttressed by an ethic of investigation and discovery. Further, this has to be done not just in relation to test execution but also in test design. That means having to listen and understand information coming from business analysts, user interface experts, developers, and other testers.
The notion of uncertainty showed an unexpected weakness in the very foundations of science. This weakness was all the more startling because it was showing cracks in an area that had previously been taken as self-evidently secure. This is something testers deal with all the time. By definition, a large part of what testing is doing is showing how much uncertainty we have about the state of an application or the state of our knowledge regarding how to build the application. That which is “self-evident” or “obvious” is often what is most questioned by the tester.
Uncertainty showed those in the scientific world that you couldn’t necessarily know everything you wanted to know, regardless of how much rigor or precision you applied. Uncertainty showed that any given “fact” could become an irreconcilable point of view. In the testing arena, this is the tessellation of viewpoints I brought up in Quality Is As Quality Does. That is part of the soul: a reconciling of the different viewpoints and encoding those viewpoints into a shared mechanism of collaboration and communication.
Another part of the soul is the balancing act. The facts of testing will be deterministic and static but the quality itself may shift and morph, often based more on human factors than on technological ones. Those human factors can be thorny issues like perception, temperament, and bias. That has interesting implications for how the “science” and the “art” of testing is carried out. Anticipating, mitigating and communicating those implications is, in my view, exactly where the soul of testing resides.
