The first and second posts in this series were about one person occupying two roles on the same project, sometimes switching between them within a single bit of work. That’s interleaving, and it works because the switch is cheap: same person, same context, same code sitting right there to look at. But most of the testing world doesn’t look like that. Most of it looks like two different people, on two different sides of a task, who never actually swap seats. I want to look at what “the role, not the person” demands in that arrangement: not interleaving so much, but cooperation. And I want to do it with something small enough to show in full.
If you haven’t read the earlier posts in this series, the short version: I treat “developer” and “tester” as roles a person occupies, not identities. The previous two posts showed what it looks like when one person switches between them rapidly, on the same code, in the same sitting. This post asks what changes when the switching isn’t rapid. When it’s two different people who each only get to do their half.
The example this time isn’t from the Z-Machine project. I wanted something small enough that the whole thing (function, test, and the reasoning behind both) fits in one sitting, with no spec document standing behind it. So: a retry function. The kind of thing that shows up in almost every codebase that talks to a network, usually written in about fifteen minutes and rarely looked at again.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
Sleep = Callable[[float], None] RandomFn = Callable[[], float] # returns a value in [0, 1) def compute_delay( attempt: int, base_delay: float, max_delay: float, rng: RandomFn, ) -> float: """Delay before the next attempt, exponential with full jitter. attempt is 1-indexed: the delay before the *second* call is compute_delay(1, ...), before the third is compute_delay(2, ...). """ exponential = base_delay * (2 ** (attempt - 1)) capped = min(exponential, max_delay) return capped * rng() # "full jitter": uniform between 0 and capped def retry_with_backoff( fn: Callable[[], T], max_attempts: int = 5, base_delay: float = 0.1, max_delay: float = 10.0, sleep: Sleep = time.sleep, rng: RandomFn = random.random, ) -> T: """Call fn, retrying on exception with exponential backoff + jitter. Raises the final exception if every attempt fails. """ for attempt in range(1, max_attempts + 1): try: return fn() except Exception: if attempt == max_attempts: raise sleep(compute_delay(attempt, base_delay, max_delay, rng)) raise AssertionError("unreachable") # max_attempts >= 1 guaranteed elsewhere |
Look at the signature. Two of those parameters, sleep and rng, have no reason to exist from a pure correctness standpoint. In production, this function is always going to call time.sleep and random.random. Hardcoding them would work. Ship it, and it retries correctly, backs off correctly, jitters correctly, every time.
So why parameterize them?
Because somebody, at the moment this function was written, was already asking a tester’s question: how would anyone verify this without an actual stopwatch and an actual coin flip? Without that question, the honest options are a test that really sleeps (slow, and flaky the moment your CI runner is under load) or a test that monkeypatches time.sleep and random.random globally, which works until two tests do it in the same process and start fighting over global state.
Notice what kind of knowledge that is. It isn’t testing knowledge in the sense of “know pytest.” It’s a design instinct (seams belong at the places where a function touches the outside world) that happens to only pay off if you already know, before you’ve written a line, that someone downstream is going to need to hold time and randomness still. A developer who has never had to write a test for code like this might reasonably not think to add those parameters. Nothing forces the thought. The tests will simply be worse, or slower, or flakier, and there will be no error message telling you why.
Now let’s consider the tests for the above code.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
def test_returns_the_result_on_first_success() -> None: calls = iter([lambda: 42]) result = retry_with_backoff(lambda: next(calls)(), sleep=lambda s: None) assert_that(result).is_equal_to(42) def test_retries_until_success_within_the_limit() -> None: attempts = [ValueError, ValueError, lambda: "ok"] fn = attempt_sequence(attempts) # helper: raises, raises, then returns sleeps: list[float] = [] result = retry_with_backoff(fn, max_attempts=5, sleep=sleeps.append) assert_that(result).is_equal_to("ok") assert_that(sleeps).is_length(2) # slept before attempt 2 and attempt 3 only def test_gives_up_after_max_attempts() -> None: fn = always_raises(ValueError) with pytest.raises(ValueError): retry_with_backoff(fn, max_attempts=3, sleep=lambda s: None) def test_computes_delay_by_the_exact_formula() -> None: # rng pinned to 1.0 (top of the jitter range) so the result is the # uncapped exponential value exactly, an oracle computable by hand. delay = compute_delay(attempt=3, base_delay=0.1, max_delay=10.0, rng=lambda: 1.0) assert_that(delay).is_equal_to(0.4) # 0.1 * 2^(3-1) = 0.4 def test_caps_delay_at_max_delay() -> None: # attempt=10 would be 0.1 * 512 = 51.2 uncapped; max_delay should win. delay = compute_delay(attempt=10, base_delay=0.1, max_delay=5.0, rng=lambda: 1.0) assert_that(delay).is_equal_to(5.0) def test_jitter_stays_within_the_capped_range() -> None: delay = compute_delay(attempt=2, base_delay=1.0, max_delay=10.0, rng=lambda: 0.5) assert_that(delay).is_equal_to(1.0) # 1.0 * 2^1 = 2.0 capped; * 0.5 jitter = 1.0 |
Specifically the two to focus on are these:
- test_retries_until_success_within_the_limit
- test_computes_delay_by_the_exact_formula
The first one only requires knowing what the function is supposed to do: retry on failure, stop on success, stop trying after the limit. You could write it from the docstring alone.
The second one is different in kind, not just detail. To write that test, whomever wrote it had to know the exponential backoff formula well enough to compute 0.1 * 2 ** (3 - 1) by hand, and had to know that pinning rng to 1.0 removes jitter from the picture entirely rather than just narrowing it. That’s not testing knowledge. That’s algorithm knowledge, borrowed. A tester who treats this function as a black box (inputs go in, retries happen, eventually it succeeds or raises) can write real tests. But they can’t write this test, and this is the one that would catch someone writing 2 ** attempt instead of 2 ** (attempt - 1), an off-by-one in the exponent that every black-box test would sail past because the function would still, eventually, succeed.
What Does This Tell Us?
Here’s what I think is the actual difference between this post and the last two. In the Z-Machine posts, I was one person switching hats mid-task: developer for a moment, tester for the next moment, sometimes both within the same design decision. Nobody had to know anything in advance; I just had to be willing to keep asking the other role’s question.
Nothing here in this post required a hat-switch. The developer who added sleep and rng to the signature didn’t stop and become a tester partway through. Instead, they wrote it that way from the first draft, because they already carried enough of the tester’s discipline to know what shape the function would need to have. And the person who wrote the exact-formula test didn’t stop and become a developer. Instead, they wrote it because they already carried enough algorithmic literacy to know what an off-by-one in an exponent would look like, and how to build a test that couldn’t miss it.
That’s cooperation instead of interleaving. Two roles, maybe even two different people, who never trade seats. But each one is carrying a working knowledge of the other’s discipline as a precondition, not an accommodation. Take that borrowed knowledge away from either side and something real breaks: an untestable function on one end, or a test suite that agrees with a bug on the other. “Developers should test better” and “testers should understand the code” are usually offered as separate pieces of advice, aimed at separate people. This function suggests they’re the same requirement, looked at from two different chairs.
