Most testers are aware of the idea of a “test case.” What people outside of testing often don’t know is how much debate can swirl around what a test case is or should be. I think it’s great to have discussions about this kind of thing but I also find that there can be a temptation to either simplify it too much or complicate it too much.

So let’s first talk about the idea of a “use case.” This is just a way of conceptualizing how something will be used. You will often hear this stated as a “written description” but it doesn’t have to be. It can be verbal. It can be a mental model in your head.

The same applies to a “test case.” A test case adds the extra burden that the use being conceptualized must be verified or falsified in some way.

Crucially, none of what I just said indicates nor demands how formal or informal the “case” is or must be. None of what I just said indicates whether the “case” must be written down or can be expressed purely verbally or simply be something that someone is thinking at any given moment, without any verbalizing or writing.

You will often hear or see some test sentiments out there saying something like this: “You don’t write tests, you perform tests.” Well, in fact, you can do both. And I think framing it as I just did here allows that flexibility.

You will also see some conversations that suggest “there is no such thing as test cases.” The sentiment here is related to the idea that since you can only “perform tests,” you can’t actually “write a test” and since, in this view, a test case must be a written thing, it must not be possible for one to exist. Needless to say, I don’t find this style of discussion very useful or practical, particularly when aimed at those outside of testing.

The level of formality and documentation related to tests are attributes. There’s flexibility in how these attributes can be expressed, whether in written form, verbal communication, or even just a mental understanding.

This, by the way, entirely matches how the basis of science and art work. There are certain things that are written down. There are other things that are verbal. There are other things that purely mental constructs. There is a mix of scripted actions and guiding heuristics.

In my view of the world, it really doesn’t have to be any more complex than what I just said above.

There can be tradeoffs and compromises to be made when you write at a high level of detail. There can be tradeoffs and compromises made when you write at a very low level of detail. There can be tradeoffs and compromises to writing anything down at all. There can also be tradeoffs and compromises to not writing anything down at all.

The fundamental point should be that any sort of test case should first and foremost be a way of thinking about something that will be observed in its use to draw conclusions about it. This is crucial because it frames the idea of thinking and acting experimentally. As long as a test case helps with that, it can be useful.

Another way to frame it is that a test case can provide a guide for exploration or an understanding for a particular path of investigation. Many such cases acting together become a powerful way of composing a total picture.

Hmmm. Composing a total picture. So, you know, this got me thinking. To my reader: I would actually stop here if the above entirely resonates with you or the above seems entirely stupid to you. In the latter case, what I say next won’t help. In the former case, it probably won’t matter. But I thought it was fun to keep thinking about this, so here we go into the rabbit hole.

What Can Calculus Teach Us About Test Cases

As I began thinking about how to frame this, the thought popped into my head about differentiation and integration in calculus.

Bear with me. It might not be as crazy as it sounds.

Differentiation in calculus is the process of finding the derivative of a function. This basically measures how the function changes as its input changes. So, if I was to construct an analogy, I might say that when you break down general ideas about how to test something into individual test cases, you are “deriving” specific ways to assess and verify different aspects of the system. Each test case could be seen as a derivative, focusing on a particular aspect of the functionality.

Integration in calculus involves finding the area under a curve. This can be thought of as accumulating or summing up infinitesimal changes over some range or scope. Again, going with my nascent analogy, when you consider a bunch of test cases together, you are “integrating” them to form a comprehensive view of the system’s behavior. This aligns with the notion of integrating test cases to provide a bigger picture of how the system behaves as a whole.

Now, yes, it’s important to note that differentiation and integration in calculus have specific mathematical properties and processes that might not have perfect parallels in the context of testing. I know this. I trust this is obvious, though. The analogy with the calculus concepts is more to show the idea of how test cases can be thought of as an individual aspect (derivative) or an entire group (integral).

Conceptually whatever we call “test cases” define a metaphorical shape to what is being tested.

This reminded me a bit of what I talked about in the shape of testing.

I think you can use those calculus concepts as a metaphor to highlight the idea that individual test cases provide detailed insights into specific aspects of a system (akin to derivatives shaping the curve) while also considering that multiple test cases together provide a holistic understanding of the system’s behavior (similar to integration creating a fuller picture under a curve).

Let’s visualize this a bit to see if I can talk myself out of the idea.

So there we have an original shape and we’ve split it apart in various ways: rings, slices, and boards. That’s often called an “x-ray view.” We can also consider another way to look at this:

Here I’m just showing how the rings accumulate over time. You build up more and more of them. These can be written test cases but they could also just be conceptual knowledge. Or a set of heuristics.

Same thing here. I just build up the boards and accumulate more of the picture.

Once again: same thing. Just building up the slices.

These are often called a “time lapse view.”

What’s a problem here, though?

Well, here’s where I have to caution against being what I called a model literalist. If you just look at the visuals, you could say that this is a very algorithmic pattern I’m showing you. I’m contributing the same amount with each step. And the parts are identical. And that doesn’t feel very much like the testing we all know and love. And I agree! On the other hand, you could also say the contribution from each step starts small and continues to get larger and that very much does feel like the testing we all know and love.

To avoid getting stuck in model literalness, let’s bring it together.

Here are rings become shells; our slices become wedges; our boards become plates. And the point is that we sort of go from a granular view to a more holistic view. And note we can go both ways. We can start with the shells, wedges, and plates and from them create rings, slices, and boards. Or we can go the other way. Or both ways simultaneously, using an explore-exploit dynamic.

What Does This Visual Tell Us?

Visualize this however you want. Maybe the sphere means we have no actual written tests, just a whole bunch of knowledge or heuristics that guide the team. But if you break apart about the sphere — slice it or time-lapse it — you end up with individual tests, even if not written down.

What you do ultimately end up with is an ever-expanding sphere of knowledge about how to test the thing you are concerned with testing. Some of which you will likely want to encode as actual written test cases. Some of those written test cases may be only intent-focused. Others might be heavily implementation-focused. And others may simply be test heuristics that guide actions. Some or all of those may be written done. Or maybe none of them are.

Believe it or not, the above is really how calculus can be conceptualized as well. The derivative is the pattern of slices we get as we apply the “x-ray view” to a given shape. The derivative just splits the shape into easier-to-measure steps. The integral is gluing together the “time lapse view” into a bunch of slices and measuring the final result.

Let’s Please the Mathematicians

I’m kidding. I’m not looking to please mathematicians. But I was actually curious if I could visualize this a bit using NumPy and Matplotlib. (At this point, you’re probably wondering: “Sheesh, how bored did this guy have to be to write this?!?”)

So let’s create a graph of a function that can represent the behavior of a system. Then, for a specific point on the graph, we could draw a tangent line to the curve at that point. This tangent line represents the derivative at that point, showcasing how it captures the instant rate of change or sensitivity of the function. In the context of test cases, you can label the tangent line as an individual test case. Maybe that illustrates how the test case provides a focused assessment of a specific aspect of the system’s behavior? I don’t know. Let’s see.

So we have a curve that represents the behavior of some system we’re testing. We choose a point of interest and calculate the corresponding point on the curve. The tangent line (derivative) at that point is determined based on the slope of the curve at that point. The tangent line is then plotted along with the system’s behavior curve.

Yeah, maybe. Kinda weird, right? Let’s try an integration example.

Here “Test Case 1” and “Test Case 2” represent two individual test cases. The “summed” test case is created by adding the values of these two test cases element-wise, representing the integrated view where both cases contribute to the overall behavior.

The dashed line in the plot represents the “summed” test case, which integrates the information from both individual test cases. What the heck is that illustrating, though? Well, the idea is to illustrate how integration, in this context, can be thought of as combining and considering multiple test cases together to provide a broader understanding of the system’s behavior.

Sigh. Not quite the visual I was shooting for. So let’s try this again with integration.

Okay, so here, we have a curve. In this case, a simple quadratic curve, although that doesn’t really matter. It just made it easier to put this together. I defined an interval [a, b] over which I wanted to calculate the area under the curve using integration. The upshot is that the shaded area under the curve illustrates the concept of integration as calculating the accumulated area. Thus the shaded region under the curve represents the integrated view or the “sum” of the values within that interval.

This demonstrates how integration is analogous to considering the accumulation of values over a range, just as I’m describing integrating multiple test cases to gain a broader understanding.

So that visual is closer to what was in my head. Can I do the same thing with differentiation?

Here the curve function remains the same. I chose a specific point and calculated the corresponding point on the curve. Then, I determined the slope of the tangent line (derivative) at that point using the derivative of the curve function.

The dashed line represents the tangent line (derivative) at the point of interest, showcasing how differentiation focuses on the slope of the curve at that specific point. This demonstrates how the derivative provides a localized insight into the behavior of the curve, analogous perhaps to how an individual test case focuses on a specific aspect of a system’s behavior.

But, honestly, it’s all more simple than that. It’s basically this:

Here I’m just showcasing the localized interaction between the test case and the curve. Each dashed line represents a test case that “touches” the curve, similar to how differentiation focuses on the slope at a specific point.

But in between each of those points can be tons of exploratory testing that the test cases guide someone along. Or the exploratory testing itself can suggest certain tests that should be written down at those “points.”

This would be an environment where, perhaps, very few tests are encoded as “written test cases” and where exploration takes higher precedence.

On the other hand, you might have more this:

Here each dashed line for a test case is now a tangent line to the curve at the corresponding point. The slope of the tangent line is determined by the derivative of the curve function at that point, just like how you would calculate the slope of the tangent line for differentiation.

This visualization illustrates how each test case is positioned as a tangent line to the curve, providing a localized assessment similar to how derivatives focus on the slope at a specific point.

This would be an environment where you have test cases for just about everything along the curve. Exploration may still play a part — and hopefully does — but the curve suggests all exploration gets encoded as “written test cases” perhaps. Maybe this is an environment that is heavily audited, as one example, or uses a lot of outsourcing for test execution.

Both views are possible. Both views have tradeoffs and compromises. The key point is: it’s a spectrum of activity based on what makes sense in your context.

Okay, So That Happened…

Yeah, so this was a post I didn’t plan on doing and that probably shows. But this was a fascinating little excursion. At least to me.

I should note that this will likely be relevant to my continuing “Thinking About AI” series since I will be providing a lot of implementation that could be thought of as test cases but, I can assure you, would not be viewed as traditional “test cases” by most testers.