There is much talk out there about whether testers should learn code. There is even more talk out there about automation. What there isn’t, at least so far as I can see, is much that shows actual examples that break down some concepts, particularly for testers entering the field. Here’s one of my attempts.

To be clear, there are a ton of examples out there showing how to use Selenium or Cypress or Postman. What I’m talking about are examples that show how developers and testers can think together. Particularly when this thinking must revolve around the one thing that we know for an absolute fact will be deployed to deliver outcomes: code! Like it or not, code is the primary first-class citizen. There are also not a lot of examples that show how looking at code can reinforce testing concepts and vice versa.

Some of what I talk about here will be related to testability and I might recommend my testability series if you’re curious about that. While that series uses Ruby as its basis, here I’m going to use Go. You don’t have to code along with me but if you want to, Go is painfully easy to install on any operating system and very easy to run consistently across operating systems.

I’ll state it again: you don’t have to actually do anything with the code in this post if you don’t want to. I’ve kept the example brief so it should be easy to understand regardless of how a reader engages with the material. I also do have a GitHub repo with tagged commits if you want to follow along that way. I’ll periodically indicate which tags to check out, should you wish.

As a tester, there is much value to learning how to engage with code and the ecosystem of a given language. There’s a reason why this series of posts is tasgged “Test Doing.”

A Simple Project

This is going to be really simple stuff: we’re going to be looking at a prime number calculator. I think you’ll find that even with something as simple as that, there’s a lot for a tester to consider, particularly in terms of where, when, how, and to what extent to test and to utilize tooling to assist that testing. To get started, let’s say your development team has created a project directory and in that directory they have code in a main.go file:

This is clearly very simple logic, right? The primary feature being built is the checkPrime function. That function will be passed some data, which is the number to check, and it will return a boolean value (true if prime, false otherwise). Also to be returned will be some outcome string that will indicate a message to the user. The developers provide a checkPrime implementation.

The business domain here is simple: a prime is a whole number that is divisible only by 1 and by itself. So, let’s run this.

go run .

If you’re playing along, but unfamiliar with Go, you’ll need what’s called a go.mod file in your directory. You can generate that by running go mod init primecalc in the project directory.

git checkout tags/start

The logic will work and note that you can change the value of the data variable in the main function to try out various data conditions. Again, pretty simple stuff. So, how much can this simple code help us understand testing? Specifically, how might this help someone new to testing who hears conflicting messages about the efficacy of working with developers, unit testing, integration testing, automation, and so on?

Simple Test Tooling

One thing to notice is that our program, as yet, has no user interface. But we don’t want to wait until a user interface exists in order to test and test regularly. This is what the cost-of-mistake curve is all about. So, let’s say, as a tester, we’re working with the developers to create some code-based tests. We’ll put this in a file called main_test.go that sits in the same project directory as the application source file.

It’s quite easy to write and run tests in Go, which removes a lot of the excuses for not doing so.

go test .

You’ll get something like this:

  ok      primes  0.330s

Notice what this test is doing. It’s essentially forcing the checkPrime function to be called with a data condition (0). Then the result and the outcome, which the function returns, are being checked. But now let’s say I want to add another condition.

Here I’m adding some comments just to keep my tests separate. Let’s consider yet one more test:

Everything works, all tests pass, but this is getting painful. It would be quite easy to make a mistake in writing these tests. Yet, there is some benefit here in that, with these tests, I no longer have to manually test for these various conditions each time by going in and changing the code. Again, I’ll note that there is no method for entry yet on a user’s part. The only way to change the input to the logic is to change the code directly.

Refining Test Expression

As a tester, let’s help the developer to consider a better way to express these tests. What do I mean by “better way to express”? Well, that can mean a lot of things. For me, right now, it means I would like a much more concise structure to indicate the actual tests and then have some logic that iterates over that structure. To do that, we can express our tests in an iterator. The above test logic can be entirely refactored.

I would recommend taking some time to understand how that refactoring improves things. These tests will still pass. But how do we know they’re actually passing? Well, that’s a great question! We should try and make them fail and see what happens! A failure would look something like this:

--- FAIL: Test_checkPrime (0.00s)
    main_test.go:22: condition: 4 | expected false, got true
    main_test.go:26: incorrect outcome
    main_test.go:27: expected: 4 is not prime; divisible by 2 | got: 7 is prime

Here I’ve purposely contrived a failure just to show you what it would look like. Speaking of test output, I should note that you can run tests with a verbose flag.

go test -v .

The output should look something like this:

=== RUN   Test_checkPrime
--- PASS: Test_checkPrime (0.00s)
PASS
ok  	primecalc	0.340s

Obviously verbosity is only helpful if you have multiple tests.

git checkout tags/tabletest

Expressing Our Conditions

Let’s consider our test conditions.

Notice a problem there? The first element, the condition, is really repeating part of the second, which is the specific data condition to supply. In fact, that first part is also repeating the last part, which is the outcome. This is actually telling us something. There’s an important point here but let’s first consider something else: coverage.

Test Coverage? Code Coverage?

You can easily get the coverage of the tests relative to the current code.

go test -cover .

You would get this output:

  ok      primecalc  0.150s  coverage: 63.6% of statements

Okay, but, how do we know what we’re not covering? Well, for that you and the developers can generate a coverage report.

go test -coverprofile="coverage.out"

What that does is generate a file of the coverage. You can look at that generated file but it’s more useful to view it in a web-based format.

go tool cover -html="coverage.out"

You should see that the only condition from the checkPrime function that is not being tested is the check for negative numbers. Just to remove ambiguity, you should likely be seeing something like this:

What we have here is our code coverage. Is this also our test coverage? Well, we’re certainly testing basis paths and thus independent paths. I’ll come back to coverage equality but, for now, let’s go back to the naming that I mentioned. Let’s change the logic so that it looks like this:

The only change here is how the first parameter is being expressed. A simple change but notice how this slightly reframes my test as showing the feature coverage at a glance. We have one “prime” test condition (with data condition 7) and two “not prime” test conditions (with data conditions 0 and 4).

Notice that wording there? Test condition and data condition. I think that’s really important for testers to get used to saying. It’s a small step to then stop worrying about so-called “happy paths” or “positive tests” or “negative tests” and instead focus on valid conditions and invalid conditions.

Should we have more “not prime” conditions? Should we have more “prime” conditions? Well, that would take us into considering equivalence classes or equivalence partitions. For now, let’s say that, working with the developers, you add the following test conditions:

Here I’m picking at the boundary of 0 but also accounting for the fact that 0 and 1 should be an equivalence class. All negative numbers form a single equivalence class. Here 4 and 9 are chosen for data conditions simply because they provide a different divisibility.

What about adding tests for a floating point number? Or a text value? Well, this is a case where Go would not even let you compile if you tried those. To prove that, you could go back to the main function and try one of these variations:

Doing that would get you one of two errors. Either this:

cannot use data (variable of type string) as int value in argument to checkPrime

Or this:

cannot use data (variable of type float64) as int value in argument to checkPrime

In my post on Test Shapes, you’ll notice the pedestal of the “test diamond” is Static, which is leveraging type checking. This is one of the earliest quality checks a developer can do! Allowing for static types and type checking can rule out entire classes of bugs.

Okay, so, with our extra conditions in place, the coverage should be at 72.7%. In fact, we are now entirely testing the code paths of the checkPrime function. In this case, that code coverage (arguably) matches the test coverage and the feature coverage as well. This holds largely because, remember, there is no user input or even interaction with this code as of yet.

Test Observability

There are few challenges here, though, with our current setup. The first challenge is that you can’t really run an individual test condition. The other is that when you do run these tests, assuming nothing fails, you just get this:

ok      primecalc  0.124s

That may actually be okay since you only want to see failures. But there’s really no way to know what was actually run, short of looking at the test code itself.

This is where patterns can help. We already used one above, which was a data table or iterator pattern. There is also a subtest pattern, which is a powerful feature that complements the table-driven testing approach. It’s not entirely distinct from the table pattern; rather, it enhances it by allowing you to run each case as an individual subtest using. In the context of Go specifically, we do this using a Run function.

It’s perhaps subtle but this change enables better isolation, debugging, and reporting of test cases. This will let you do this:

go test -v -run=Test_checkPrime/not_prime

That will only run the “not prime” tests. However, consider this:

go test -v -run=Test_checkPrime/prime

You’ll see that this command runs both prime and not prime tests. This is happening because the -run flag in Go’s test command uses regular expressions to filter which tests to run. To ensure you only run the exact test case labeled “prime,” you can adjust the regex pattern to match it precisely:

go test -v -run=Test_checkPrime/^prime$

Another way that I think is better would be to adjust the test labels to avoid substring conflicts.

Here notice I changed only the last condition’s first parameter. Now you can run both test conditions without the need for any regular expression syntax:

go test -v -run=Test_checkPrime/is_prime
go test -v -run=Test_checkPrime/not_prime

There’s yet one more problem. With the current output for the “not prime” conditions, you get this:

=== RUN   Test_checkPrime
=== RUN   Test_checkPrime/not_prime
=== RUN   Test_checkPrime/not_prime#01
=== RUN   Test_checkPrime/not_prime#02
=== RUN   Test_checkPrime/not_prime#03
=== RUN   Test_checkPrime/not_prime#04
--- PASS: Test_checkPrime (0.00s)

Notice how we just have numbered elements? That’s not every helpful. We can construct a simple test name and use that as the output:

Your output will now look something like this:

=== RUN   Test_checkPrime
=== RUN   Test_checkPrime/not_prime_-1
=== RUN   Test_checkPrime/not_prime_0
=== RUN   Test_checkPrime/not_prime_1
=== RUN   Test_checkPrime/not_prime_4
=== RUN   Test_checkPrime/not_prime_9
--- PASS: Test_checkPrime (0.00s)

That’s at least a little better.

git checkout tags/subtest

Putting Pressure on Design

I’ve talked before about how tests put pressure on design. But that doesn’t have to be so. Let’s consider two other things that our tests really wouldn’t have told us directly. To do that, let’s consider two aspects of the original code that’s being tested. Here’s our function again:

In looking at our tests with the developers, it might become clear that the first two “if” conditions could really be wrapped up in one:

This combines the checks for data == 0, data == 1, and data < 0 into a single, clean condition. It improves readability by reducing redundancy and makes it clear that any data condition that is less than or equal to 1 is not prime. However, you lose some specificity in the error message, at least around negative numbers.

Once this change is made, the power of our tests is that we can just rerun them and make sure we didn’t break anything. And if we ran that we would find that our test fails:

--- FAIL: Test_checkPrime (0.00s)
    --- FAIL: Test_checkPrime/not_prime_-1 (0.00s)
        main_test.go:40: FAIL: not prime - checkPrime(-1) message was -1 is not prime; expected negative numbers are not prime

That failure makes sense because we no longer actually have the specific “negative numbers” message. So the outcome from our test condition changes to this:

If you’ve been playing along, you might notice that your coverage drops from 72.7% to 66.7%. Why? Well, consider that there’s less code actually being executed right now. The coverage percentage is the lines of code executed by the tests divided by the total lines of code multipled by 100.

Let’s also consider that loop over the primes. One of the developers suggests that this can, and should, be changed from this:

To this:

This is a more significant optimization. The reason this works is that if a number n has a divisor greater than sqrt(n), it must also have a divisor smaller than sqrt(n). By only iterating up to sqrt(n), we eliminate unnecessary checks and improve the function’s efficiency for large numbers.

As before, once the change is made, our automated tests make re-testing all of our conditions trivial. Obviously our coverage would remain the same.

None of our tests would have necessarily pointed that change out to us. Whether we look at code coverage, test coverage or feature coverage, none of those necessarily told us about the internal quality of performance for our code. Yet, they did tell us about a way to streamline our conditions and remove some complexity.

The main thing to note here is that we improved our testability by reducing the surface area of our code, thus perhaps improving maintainability, but also we potentially improved its performance.

What this tells us is that tests can put pressure on design but that’s not a guarantee. And notice that this would apply here whether we wrote our tests first, concurrent, or after relative to writing the code.

Whether someone wants to call any of this “test driven development” is largely up to them. More importantly is that we created a fast feedback loop by putting tests very close to where we make mistakes and thus reduced our cost-of-mistake curve.

Test and Code

Notice how much ground we covered here and this was for a relatively simple bit of logic that doesn’t even have the basis of user interaction. This is the stuff that people coming into testing need to see and understand. I would argue that this is the stuff that some long-timers in testing also need to understand.

We explored code and we explored using automation to complement our test execution. But, more crucially, we explored test concepts: basis paths, equivalence classes, boundary values, test conditions, and data conditions.

These are the discussions testers need to be having more of in my opinion.

Rather than experienced testers arguing over whether and how much testers need to learn code (they do!), they should show how learning code helps understand and frame those test concepts I mentioned. Rather than experiences testers engaging in marginally useful debates about how automation isn’t testing (it is a form of testing!), they should be able to show — not just tell — how automation has its place but also has its limits.

In the next post, I’ll add some user interaction to this code example and see if that complicates things.