I’ve talked quite a bit about the interview process for testers. Here I’ll try to distill some of that material around my experiences with hiring test specialists. By this term, I mean exactly what it sounds like: people who have chosen to specialize in the discipline of testing.

Testers focus on experimentation, investigation, and exploration. These are part of the technical discipline of testing. This is how testers practice their craft. It’s very important to avoid equating “being technical” with programmatic skills. The latter are just one aspect of what a “technical tester” means. So, again, don’t confuse “technical” with “programmatic.” Don’t even confuse “technical” with “solving algorithms.”

When talking to a tester, you want to find out where they exist on the spectrum of being an experimenter, investigator, and explorer. In my experience I have found this spectrum will fall along the following lines:

• practice-driven –> principle-driven –> intuition-driven –> theory-driven

A simplistic way to look at the spectrum is to consider it as going from “Junior” to “Senior” but it’s often not that simple. Sometimes you’ll have someone that is theory-driven but is unable to do some of the practice. Or find someone with the right intuitions but unable to connect those intuitions to principles on the one side and theory on the others.

If that’s the case, how do we ask questions along this spectrum?

Well, to know which questions are the right questions for finding specialist testers, you have to know what’s important and what matters. And you can’t know that unless you know what outcome is desirable for you. What kind of person, specifically, do you want? How much of a balance between an “automation engineer” and a “test engineer,” for example? That, coupled with the candidate’s experience level in the industry, should guide where and how you focus along the above spectrum.

In large part testing, as a craft, is about variably-predictive decision making under conditions of uncertainty. This means testers should be capable of dealing with a certain amount of randomness, unpredictability, opacity, and incomplete knowledge. Not everything goes according to a script in testing and that variability should be reflected in how we seek out the candidates that best exemplify the discipline and craft of testing.

So let’s talk about the first time you’re likely to engage with the person.

Let’s Talk! (The Phone Screen)

During an initial phone conversation, it should be more than possible to decide where on that spectrum someone currently lies. The easiest way to do that is to provide a scenario made up of related parts that encourages the candidate to shift among polarities: working with business, working with developers, working with testers, writing human tests, converting human tests to automation, etc.

It’s important in these scenarios that each question is a follow-on to the response they gave you, not necessarily the next question you planned to ask. You have to adapt just as much as the candidate does because fluidity of thought and the ability to move between abstractions is one of the core skills of a tester. So if your company is one that focuses on a “list of questions to get through”, you are doing it entirely wrong.

It’s also important that the candidate is challenged a bit. For example, pose challenging questions that the scenario guides you to:

• “Why is randomness bad in testing?”
• “Why does having a test plan compromise how testing is done?”
• “Hmm, so testing doesn’t really seem all that interesting. Why do you do it?”

Note that you may in fact disagree with what you are asking the person. For example, you may feel some randomness is, in fact, good in testing. Maybe you don’t feel having a test plan compromises anything. That’s fine. What’s important is not whether the person agrees or disagrees with you but how they articulate their own viewpoint.

Let’s Meet! (The On Site)

When a candidate is brought in, it should be because you have sufficient reason to believe they are a good investigator and experimenter. If programmatic skills are desired as well, you should have sufficient reason to believe that they understand enough about coding in the context of testing that they would be comfortable looking at and reasoning about code in an interview session. On the investigation side, candidates should be given a chance to show that they have very good detection methods for the following:

• Inconsistency
• Ambiguity
• Contradiction

This should be something they can demonstrate to you, not just talk about. This is important. Ask yourself if you’ve thought about how you will discover this about a specialist you are interviewing. If you haven’t thought about this at all or don’t have a means of allowing the specialist to demonstrate this, how are you sure you’re getting a specialist? On the experimentation side, the candidate should be given a situation where they can display the behaviors of a good experimentalist. Specifically:

• They need to show how they determine if the experiment is going well.
• They need to show how they determine if the experiment is going poorly.
• They need to show how they amplify the parts that are going well.
• They need to show how they dampen the parts that are going poorly.

Even more specifically, what we want to learn from the person we are talking to are answers to these kinds of questions:

• How do you gather evidence?
• How do you reason through a problem?
• How do you decide what evidence matters and what does not?
• How do you conduct experiments to get observations?
• How do you decide what observations matter and which do not?
• How do you decide where to focus your time when time is limited?
• How do you reconcile differences of opinion regarding evidence or observations?
• How do you reconcile different interpretations for the same evidence?
• How do you avoid soft reasoning and shallow questions?
• How do you balance exploration of the unknown with exploitation of the known?
• How do you use an always/never heuristic in testing?
• How do you articulate all of the above?

Again, if you don’t have a means of figuring this out, how do you know you’re getting a specialist? If you don’t have an interview process in place that helps discover the above, your notion of what a test specialist is likely extremely compromised.

If programmatic skills are desired, the above discovery on your part should also include learning from the person the answers to these kinds of questions:

• How do you turn human tests into code?
• How do you recognize good patterns when tests become code?
• How do you recognize the parts of human testing that should not be turned into code?
• How do you recognize bad coding in the context of tests?
• How do you recognize too much complexity in test automation?
• How do you recognize when technical debt is being introduced into test automation?

Note that this is geared towards figuring out the programmatic skill of the specialist in the context of testing. So if you are asking a candidate to solve algorithms, you are doing it wrong. You are likely looking for a programmer, in that case. Someone who has specialized in programming is different than someone who has specialized in testing, even if that speciality has provided them some programmatic skills.

Sync the Phone and the On Site

If you are bringing the person in to meet with a wider team, then that implies that on the phone the person gave you enough indication to bring them in. And that was likely due to talking about what they have done, what they can do, etc. It’s critical that the on site be used to give the person a chance to demonstrate what they talked about.

Even if part of your interview process has a “homework” challenge of some sort, make sure the homework challenge provides you with enough actionable information to determine if you are dealing with someone who has specialized in testing. This is distinct from “someone who can test” or “someone who can program.”

Agile Testers?

A lot of people and companies are out there looking for so-called “agile testers.” The distinction is largely meaningless. But let’s go with this for a second.

A recent book called Thinking-Driven Testing takes the idea of a manifesto and applies it as such:

A professional tester values:

• critical thinking over procedures and rules,

• skills over tools,

• creativeness over following the process,

• systems thinking over working in isolation,

• initiative and courage to change over ‘business as usual’.

That is, while there is value in the items on the right, a professional tester values the items on the left more.

If there was a manifesto — and I’m not a fan of manifestos — I would like to just condense it to something like this:

• Prefer adaptability, exploration, and creativity over planning, objectives, and certainty.

A specialist will be able to demonstrate a preference for the former over the latter. It doesn’t mean that they don’t plan, or are incapable of working towards objectives, or won’t provide conditional certainty. But it does mean that they place more emphasis on adapting, exploring and being creative with how testing, as a discipline, is integrated into your company.

This is even more important if your company is one that is “trying to be agile” or “adopting a DevOps culture” or “shifting more to the left” or “moving towards services” or “focusing on a continuous delivery and deployment strategy.”

Do You Really Want a Test Specialist?

Understanding what that preference I just stated means and applying it to someone who has specialized in testing requires you carefully considering whether you, or your company, actually wants a specialist tester. And that requires understanding what a specialist tester actually does and how that person adds value, beyond just “they check if my product works” or “they can write a lot of automation.”

I’ll talk about whether or not specialist testers should even be hired in my next post on this topic.