I’ve heard a lot of people in the software industry say that “testing is more art than science.” My opinion is that people who say this most likely know little about art and probably know even less about science.

Wendell Bell, in his Foundations of Future Studies: History, Purposes, and Knowledge, discusses the supposed “cognitive split” of art and science that people bring up and, in his words, “the problem is that the distinctions are false.” I agree in that I believe such distinctions are a false dichotomy. More specifically, there seems to be this notion that there’s some hard-set, always-clear division between the emotive and cognitive modes of thinking. Some believe that this division defines the principle differences between art and science. After doing a lot of reading about both, I believe that if you attempt to straddle this “division” you find that, in fact, there are many more similarities than there are differences.

Other Stuff to Read: For further reading you might check out the excellent Art and Science of Logic by Daniel Bonevac. Two other books (both by Jacob Bronowski) that I really enjoyed are The Visionary Eye: Essays in the Arts, Literature, and Science and The Origins of Knowledge and Imagination. You might also want to check out Nelson Goodman’s essay “The Activity of Aesthetic Experience”, published in the Contemporary Philosophy of Art: Readings in Analytic Aesthetics.

Okay, so moving along, keep in mind this simple idea: hypothesis is the art of science. Now, even if this statement allows me to say that testing is art and science, I think people will more want to know about how the science part fits into that than the art. So let’s consider science for a moment here.

Science is an access route to knowledge. Science is a rigorous discipline and, in many respects, it’s an unforgiving discipline. Science tends to be unsentimental in that it discards old ideas on the trash heap of history as fast as they are discredited and that means it has a strong measure of objectivity when performed well. As Aldous Huxley so adequately put it:

The great tragedy of Science – the slaying of a beautiful hypothesis by an ugly fact.

Science accepts new concepts only insofar as they explain the facts at hand better than the old ones. Science is not necessarily the place where answers are found; rather it’s the place where questions are asked. If you look at it one way, science isn’t about knowledge — it’s about ignorance, because science is about nothing if it’s not about the search for greater knowledge. And such a search implies we are lacking in our knowledge; that we have more to learn. In science, an answer is only the place where you stopped asking the questions. Konrad Lorens said it well when he said:

Truth in science can be defined as the working hypothesis best suited to open the way to the next better one.

The core of science is the scientific method and the basis of the scientific method is to serve as a procedure for determining the difference between knowledge and belief. Belief is kind of like wishing, in many ways. Belief exists inside a person and is often based on personal conviction rather than external referents. Belief is a conviction that some system works a certain way, even though you may have no actual evidence at all to prove it and may even have contrary evidence.

Knowledge, on the other hand, is evidence. Knowledge exists outside yourself. Knowledge is measurable, it’s testable, it’s repeatable, and it’s demonstrable. Most important, knowledge can be communicated to other people. They can repeat the measurements and tests and demonstrate the same facts for themselves. In other words, knowledge is what we can all agree on, because we can each verify the alleged facts for ourselves. So the scientific method is a procedure for determining the evidence. It’s a way of establishing agreement. George Santayana said it well:

Science is nothing but developed perception, interpreted intent, common sense rounded out and minutely articulated.

Here’s how this method works. You observe a phenomenon of some kind – a condition, a behavior, a curiosity – something that you don’t understand or that you want to understand better. Other people may observe it too, which helps because then you know it exists as opposed to you just believing it exists based on perhaps faulty perceptions. Still, initially you probably will have no immediate explanation for what you’re seeing. So you postulate a hypothesis. Is your hypothesis accurate or not? How do you find out? You test it. You design an experiment that (ideally) allows you to test only that single question, and nothing else; in other words, the only variable is the one you’re testing and everything else is a constant from one test to the next.

Here’s a key point… Whatever the specific result of the experiment/test, it’s always a success. Why? I say this because the test always gives you information. You either get information about what works or you get information about what doesn’t work. Either way, the result of the test is a fact and each fact is a piece of knowledge.

Even if the test confirms your hypothesis, you’re still not done. In fact, in science you’re never truly done. One of two things can happen: either a better hypothesis comes along and you throw out the old one, or another fact comes along that doesn’t fit your hypothesis and you’ve got to come up with a new one or amend the existing one.

So the scientific method is about asking whatever question you have … and then asking it again … and then asking it yet again. This method is about testing the facts as well as the hypothesis, pretty much forever and even a few days after that. Hypotheses are never really proven; they’re simply used – and only as long as they’re useful. Albert Einstein said it well:

Truth is what stands the test of experience.

If your hypothesis lasts for quite some time and seems to hold up well, you get to call it a theory. Even then, however, it’s still technically provisional. The theory is just the best answer we have right now. Perhaps the theory is the only answer; perhaps not. You can’t really know. All you can say is whether that theory withstands the test of time and whether further observations align with the theory; and whether new facts seem to contradict the old facts.

Keep in mind that a theory is a map of a specific terrain of knowledge. The theory is not the territory; it’s just a representation of it. Depending on what you need to do, you specify what kind of a map you need. As you continue to refine the different maps of the terrain you’ve discovered, the various maps become more accurate and more useful and consequently you become more powerful in terms of what you can accomplish. That’s why we make maps and that’s what science (and testing) is really about.

Now, I haven’t really specifically compared and contrasted the scientific method with a “testing method.” That may be for a later post. Further, I’ve not indicated how the role of testing fits into each description I gave above regarding the nature of science. For now, that’s left up to you, the individual reader, to discern for yourself as you think about the issue. After all, that’s what science (and art) make us do: think.