I see a lot of companies who have trouble getting started with quality assurance and test positions. They do a lot of interviewing but make a lot of mistakes when bringing in those crucial first people that will let them scale for the future. These companies look for things like “ability to write test cases” or “knowledge of automation.” They don’t look for people who have specialized in testing. But what does that even mean?

After all, wouldn’t knowledge of automation and the ability to write tests be part of a specialist skill set. To an extent, yes. But to a very small extent. After all, most people can write tests of some sort. Any programmer, as opposed to developer, can likely write automation

That said, not all people can write good tests that explore around problem areas where our cognitive abilities fail us when we design complex things. Not all programmers can write automation that operates at appropriate levels of abstraction to put just as much emphasis on readability as on maintainability.

To make sure my context is clear here, I’m mainly talking about those companies that are hiring their first substantive role in quality assurance and testing. But this equally applies to those that have hired very junior people in the past and are now trying to scale up a bit.

I see a lot of companies who look for a “junior” role for quality assurance and testing, assuming that they will just “get something done” in this area and then they can hire more senior people later. This is often — not always, but often — a terrible mistake, particularly if the juniors are your first hires. Let’s leave aside for now what “junior” even means relative to a “senior” because that alone is an area fraught with peril in the form of conflicting opinions.

I also see many companies who routinely take far longer to search for their “Test Leads” or “QA Leads”, often going through round after round of interviews, rarely making a decision. And then, when they finally do, it’s on a candidate who ticked off the checkboxes of what they thought testing meant for them (an execution mechanism) rather than what it needs to be (a scaling mechanism).

That last point is really important! Are you hiring for testing as a means of executing or as a means of scaling?

What this all comes down to is I see companies planning to hire their test specialists later rather than sooner or, ideally, first. This is particularly problematic and sad to see for those companies that have a chance to get it right early on, but fail to do so. I feel comfortable stating it so harshly because a large part of my career has been spent helping companies to course correct for the mistakes they made. To be fair, however, I can understand why it’s not clear that they are making the mistakes at the time.

I’ve already talked about asking the question of whether we should hire test specialists. I’ve also talked about finding those specialist testers as well as hiring test specialists. So here I’ll approach this from a slightly different angle.

Start With the Premise of Quality

I recently re-read the book The Advantage and a key point stood out to me:

“An organization that is healthy will inevitably get smarter over time … In contrast, smart organizations don’t seem to have any greater chance of getting healthier by virtue of their intelligence.”

A simple point, but one worth reinforcing. It’s yet another example of parallax that we often get incorrect in our industry. To reinforce this point, consider how we often hear something like the following:

• If we just have automation, we’ll have better testing.
• If we just have better testing, we’ll have better quality.

When, in fact, you have to start with the premise of quality first.

• What does it mean to add value?
• What does it mean to provide an meaningful experience for our users?

You start with those questions — as opposed to starting with statements — and you learn how to develop your testing. And when you’ve learned how to develop your testing then, and only then, can automation help you scale.

So what does this mean when you bring in people to start up a test team from scratch? Does it impact how you interview them?

This is the challenge because those doing the interviewing for these positions are those often least capable of truly assessing the needs of their testing and quality assurance. Instead, they are going on “what the industry says,” which is conflicting at best, or on their own experiences, which are also often conflicting at best.

So let me turn this around. Rather than look at this from the standpoint of the hiring manager or the hiring team, let me frame it around myself. Specifically, how do I go into interview where I consider myself a test specialist and I’m going for the position. This is where I make sure people understand a few things that I very much believe.

Distribute Quality

Quality is a shifting perception of value over time. Regardless of what “the industry” says or does, quality assurance is not a team or a person; it’s a distributed function among a delivery team. That function is one that makes malleable determinations of what adds the most value in a given context for the most reasonable cost over a particular duration of time.

Testing is a way of a discovering information and putting ideas under the crucible of experimentation, some of which can be supported by tools (automation) and some of which must be performed solely by humans operating in the context of understanding what other humans value, why they value it, and how that value shifts over time, space, and mood.

That understanding is what a test specialist brings you and sets you up to scale. If you are an interviewer and don’t understand why the above, as a mindset, adds value, you shouldn’t be making the decisions for hiring test and quality assurance specialists. For example, from the above you could ask some questions:

• “Okay, so how do you go about discovering that information?”
• “What do you mean by experimentation?”

A challenge here is the act of “assuring quality” is a distributed one. This is something that many teams don’t seem to get. They may even say it and agree with it intellectually; but it’s often not something they’ve internalized.

A key point for the interviewer that I want to make sure is clear: you’re not hiring someone to “start your quality assurance team.” If you are, you’re doing it wrong. And if the person you’re interviewing doesn’t tell you that, you’re in danger of hiring the wrong person.

A test specialist understands that quality is part of the story we tell ourselves about how we deliver value to people. A delivery team as a whole all has input into that story, and so any notion of “assuring quality” is, by definition, distributed. And even then it’s important to recognize that this is always provisional assurance. It’s critical that the first people you bring in have a view of making sure teams are thinking like that.

Yes, tactics are important. How that is all done is important to determine during an interview. But strategy is equally important. What is being done and why it is being done are also important to determine during an interview.

A focal point of my career has been recognizing that quality is a shifting perception of value over time. Part of what all of us ultimately do is provide experiences for users. We have to do that in the context of a tessellation of viewpoints about what “quality” actually means. And sometimes that viewpoint shifts based on mood. That feature that you loved the other day because it made you think the app was so secure ticks you off the next when you’re in a hurry and the secure part is slowing you down. So we have the exact same feature, functioning exactly the same way … but the perception of quality has changed.

And that gets into design first and foremost. Design is about keeping change as cheap as possible at all times. As such design refers to the forces that exert sudden and unexpected pressures on our work.

Forces and pressures are what lead us to take shortcuts to quality or, if not shortcuts, at the very least, we simplify quality way more than we should, equating it simply with “functionality” rather than “experience.”

If you start off your testing and quality assurance with people who don’t bring in these habits of mind and patterns of thought, you are effectively bringing in people that will build you into holes you will need to dig yourself out of. It’s those very holes that have led to testing being such a variably appreciated discipline over time, lately being turned over to a technocracy.

Democratize Testing

I’ll often tell people during an interview that my belief is everyone tests, but not everyone is a specialist tester. In fact, that’s not a belief. It’s a fact. You can’t be a thinking human being without doing some sort of what we call “testing.” But just as all of us could chip away parts of a rock, that doesn’t make as a specialist in the act of sculpture.

Practically speaking, specialist testers are able to harness the individual bits of testing that all team members do — at various abstraction levels — and are able to help the delivery team maintain and persist a shared understanding of what quality means for whatever it is that’s being released. This actually applies whether you are “being agile” or “doing waterfall” or some combination thereof. This applies whether you are releasing once a day, multiple times a day, once a week, every two weeks, and so on.

This means you have used testing as a project force — specifically, a design pressure. You are using testing to put pressure on design by making sure testability is a key quality metric. This also means you have created a cost of mistake curve that provides for a very short feedback loop and where the feedback that you get is empirical enough to help people reason about what they are building and thus make better decisions sooner.

The Specialist Focus

Specialist testers need to be good at articulating all of the above. They need to be even better at implementing the above.

What does that mean for modern testers who are often shoved into the technocracy, asked to devote any and all efforts towards “more and better” automation? Especially when this is reflected in the interview process?

The “SDET culture,” if there is such, is about turning testing into a programming problem. That’s where much of what I’ve said above is presumed to be subsumed under various forms of automation. The “agile culture” — again, if there is one such thing — is often used more to turn testing into a management problem; and by “management” here I mean a logistic problem as in managing the flow of activities to get something out the door.

Both of those cultural aspects — testing as management problem and testing as programming problem — can be (and often are) entirely orthogonal to the conscious democratization of testing among a team. I say that because many people (especially testers) still equate testing with being solely an execution activity. That is what reinforces SDET and agile issues.

There’s an old joke: “Once upon a time, a politician, a statistician, and a mathematical physicist were riding on a train through Scotland. They all looked out the window and saw a black sheep. The politician said, “Look, Scottish sheep are black!” The statistician said, “No, one Scottish sheep is black.” The mathematical physicist corrected them both: “No, at least one side of one Scottish sheep is black.”

The point of this mathematical attempt at humor is to clarify what methodology we wish to use. The method of a politician (generalizing from specifics) will certainly get it wrong. On the other hand, the temptation to limit ourselves to some sort of mathematical precision (talking only about one side of one sheep) will render us virtually unable to say anything, if not make us look just plain timid. After all, the odds are good both sides of that sheep were black, right?

We see this reflected in testers who are either too willing to come to conclusions or too unwilling to come to any conclusions. Specialist testers understand that the middle way of interpreting “facts” — which is still different from the statistician, who was simply less mathematically precise — is not a science but an art.

Was There a Point Here?

My goal here was not to articulate every aspect of how a hiring manager, or hiring team, would know they are getting a test specialist.

Right now, my only goal was to provide my viewpoint that there is a problem with at least some surrounding context for that problem. A future post will talk about what starting up quality assurance or testing actually looks like.