The Quest for Testers

I seem to be on a rant lately about interview techniques for getting good testers. Here I’m going to back up a little further what I do in order to find effective and efficient testers.

I recently went through some interviews with Amazon and, while this may sound a little arrogant, I was terribly disappointed in their interview process. Great, fantastic company; very substandard interview process. I say that because it was so … by the numbers. It was so “traditional” and predictable. I’m not sure exactly what I expected, but I guess I expected more creativity in the process.

But then again: who am I to talk? What do I do that’s so different or creative?

Fair question. So first I’ll talk about my own thoughts regarding some core high-level skills of testers. It is these skills, I believe that you have to ferret out, because they are the hardest to train for. After that, I’ll talk about a particular interview challenge I give to candidates that I’ve found to be very effective in ferreting out the skills I really want.

The Tester, Sub Specie Aeternitatis

Above and beyond all, what is of paramount importance for me is that testers are motivated, engaged, curious, and inspired to solve problems, draw conclusions, and generate new knowledge. Everything else can be trained for during employment. But all the stuff I just listed can be hard to “train” on. Instead, it should be what the person brings with them — at least to some extent.

Given some time in the industry, the most effective testers should have a precise, expressive testing vocabulary for their philosophy of testing. This means testers should have the ability to speak in precise non-technical and technical terms about their testing, as well as to articulate a story about their thought processes. They should be able to demonstrate that to you during an interview challenge. Even if the candidate is relatively “junior” in terms of time within the industry, the candidates that have an instinct for testing will have an intuition for balancing preciseness with expressiveness.

Testers have to be good at moving, step-by-step, to a conclusion, and to move through a series of conclusions to a specific result. Testers have to demonstrate how different conclusions at any step might change the result. This is best demonstrated through interview exercises that are based on exploration.

There are a few thinking fallacies that we human beings all fall prey to and these are things I like to test for in testers. Let’s consider some specifics.

  • People tend to make naive observations of the past and treating those as definitive and representative of the future.
  • People make the error of confirmation. Meaning, we focus on pre-selected segments of the seen and generalize to the unseen.
  • People will perform distortion of silent evidence. What we see is not all there is, but sometimes we treat it as if it was.
  • People commit the narrative fallacy. This means we fool ourselves with stories that cater to a distinct pattern.
  • People commit the tunnel fallacy. Meaning, we focus on a few well-defined sources of uncertainty or on too specific of attributes.

All of this is what I like to check for, at least to some degree during the interview process. Rather than do code interviews for testers — which do have their place, but not usually in the simplistic way they are done — I try to provide exploration exercises that allow me to check for a series of traits that showcase thinking skills and problem solving skills while also being able to articulate at different technical levels.

Interview Challenge: Test Quest

I’ve done a lot of independent game testing for companies (see Testing Games is Hard!) and I’ve often found that testers who had a good mentality for testing games made some of the best testers in other venues as well. So wrote a game called Test Quest and I provide that to candidates as a means of looking for the traits I mentioned above. The game requires DosBox, which is available for multiple platforms.

The game is a one-room game. So this isn’t something where you are going to have to require the candidate to spend a long amount of time exploring the problem space. The game has a character that can be moved around via mouse or keyboard.

See the little guy there? That’s the character you control. The game also presents a typing interface for commands.

The game does have an interface to control the game, via menus as well:

The challenge is presented to the candidate as such:

“We’re competing in the casual gaming space, which is quite demanding with little room for bad games. In order to meet some upcoming gaming competitions for ‘short challenge games’, meaning they can be solved in about five minutes or so, we want to release our game “Test Quest”. We need to ferret out the bugs and also determine if the game is at least challenging enough but without being too challenging. I’m totally new to testing and you’re the experienced person so I’m turning to you for help. I’d like to pair test with you.”

The candidate is given a set of requirements for the game, which include some design notes, as well as a game manual that users of the game will be able to download. With all that, they’re off to the races. These materials are kept short to make sure the interview session does not just focus on them. The candidate is most certainly allowed to ask questions about any of the material provided.

One thing I should note is that the candidate is given a walkthrough for the game as part of the requirements. That’s a key point. I’m not expecting the candidate to be a game player and I’m certainly not expecting them to solve puzzles in a game. Further, this game is not based on reflexes or any major hand-eye coordination nor is there a time limit, save that of the interview time.

What I am expecting the candidate to do is explore the game with a tester mindset. And by the end of the interview period I ask them one question:

“So do we release the game or not?”

Based on what I just observed them do over the course of the exploration and testing of the game, it helps me determine upon what evidence they utilize to come up with their answer. I should also note that I don’t just sit quietly watching the candidate. I encourage them to “think out loud” and I play the part of a fellow tester, developer, or business analyst depending upon whom the candidate wants input from. I keep a running discussion going during the interview.

I also try to have a little fun with the candidate, joking a bit as we explore and test the game together. After all, if we have to eventually work together, I want to make sure that we get a good feel for each others’ personalities.

I expect the candidate to explore all of the provided material — the documentation, the requirements, the game — and simply show me how they interact with a problem space — particularly one they’ve never seen before — and eventually come to make decisions about that space based upon the testing they perform under time and resource constraints.

I also do have the source code of the game provided if candidates would like to take a look at it. It’s written in a Lisp-like language so as to minimize the chances that candidates familiar with more traditional languages will have any sort of advantage. Here’s an example of what the code looks like:

Even not knowing Lisp someone could probably get some idea of what is likely happening there.

There’s quite a bit more to say on this but I can’t give away too much information about the game because it’s still used at a variety of companies. Eventually, however, I’ll be writing a new type of game and I’ll be able to reveal more about this one, including making it available for anyone who wants it.

What Does This Tell Me?

I trust that even with this lack of a deep-dive into the game itself, you can see why this might be helpful as an exercise. As you can probably imagine the game is designed with some bugs: some obvious, some not so obvious. (The third screen shot above shows one possible bug. Can you spot it?) When and if the candidate stumbles across those, I ask them what kind of bug report they would write. At some points I’ll ask the candidate how they would write test cases or a test approach for the game.

The game also has a few elements that are designed to be a little frustrating. How someone deals with that matters to me. The game also has a few “traps” where a candidate could end up spending a whole lot of time, to the exclusion of time spent elsewhere. This helps me see how the candidate prioritizes their activities when they know time is limited. Given the initial scenario I start them off with, it also helps me see how they treat areas of an application that they consider risky.

Clearly the candidate is getting a lot of input thrown at them. There’s me verbally talking with them, there are the documents they are provided with, and there is the game itself. So what I want to look for in all this are some traits that I believe good testers must have:

  • Be thorough in reading.
  • Be patient in listening.
  • Be careful in questioning.
  • Be thoughtful in evaluating.
  • Be precise in checking responses.

Further aspects that I’m looking for:

  • Showing precision of thought.
  • Being articulate and persuasive.
  • Identifying the salient points of an issue quickly.
  • Having an evidential bias that demands proof.
  • Separating emotion from analysis and decisions.

That last point is important to me. I want to see if candidates avoid emotional ties to a particular position. For example, some people get caught up in the “fun” of playing a game. This gamer position then takes precedence over a tester position. Some people get frustrated with the requirements or the game interface itself and begin to feel that the game is “not testable” or “should not be tested” given its current state.

Also of import to me is that I want to see if a candidate can accept a certain amount of ambiguity and use it as flexibility for creative and innovative approaches to testing.

What I ultimately want to see is that I’m dealing with a candidate that …

  • … manages with data
  • … supports with facts
  • … convinces with evidence
  • … makes appropriate operational distinctions
  • … does not hide complexity behind a facade of simplicity
  • … does not mask simplicity by a falsely assumed complexity

I want to see candidates that learn to parse a question (“should this game be released?”) down into its component parts, to carve away the emotion and the confusion and leave only the thing they are dealing with. I want to see candidates that don’t get distracted by the problem until they understand the nature of the problem and what aspects of the problem need to be decided now or can safely be deferred until later. I want to see candidates who are comfortable with the “unfairness” of intransparency on some projects and use that uncertainty to think about the allocation of risk.

Testers need the ability to ask the right questions, investigate the right issues, put together the pieces of the puzzle, and draw the right conclusions. They are thus curious, experimental, and analytical. Testers have a mode where they experience the joy of exploration (and discovery) coupled with the pleasure of the hunt.

Test Quest is my attempt to seek out those abilities in candidates.

Share

About Jeff Nyman

Anything I put here is an approximation of the truth. You’re getting a particular view of myself … and it’s the view I’m choosing to present to you. If you’ve never met me before in person, please realize I’m not the same in person as I am in writing. That’s because I can only put part of myself down into words.

If you have met me before in person then I’d ask you to consider that the view you’ve formed that way and the view you come to by reading what I say here may, in fact, both be true. I’d advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

This entry was posted in Career. Bookmark the permalink.

10 Responses to The Quest for Testers

  1. Ken Hulan says:

    This is absolutely brilliant! Honestly, I’m stunned. I never even considered doing something like this but I can absolutely see the value.

    In terms of the interviewer, have you found there’s a danger of being subjective? What I mean is that I might play the game with a candidate and another interviewer might do so, but we both apply subjective measures of how well the candidate is or isn’t doing. I guess my question is whether you’ve found that you can apply objective measures, not just across candidates but also across interviewers. I’m assuming you have, since it seems you put a lot of thought into this.

    Truly great stuff. Now you got me wanting to do something similar.

    • Jeff Nyman says:

      I do provide a “response gauge” to interviewers so that they can judge some aspects of how the candidate is doing.

      Some things are obvious and more or less objective. For example, I’ve had some candidates barely do anything at all. They don’t even get started exploring the game. That’s clearly a red flag! I have others that explore and explore and … explore. But they never really settle into testing. That’s also a red flag.

      So those are elements that any interviewer can look for and make judgments upon that will likely be in line with the judgments of any other interviewer.

      There are very specific bugs in the game. Some are harder to spot, others not so much. So the extent to which candidates find those and recognize them as bugs can certainly be measured. Sometimes different candidates have a different idea of what is and is not a bug, and that’s fine. I don’t so much ask that the interviewer agree with the candidate, but simply note if the candidate finds things and has a stance one way or the other on what they’ve found. So in these cases, different interviewers will apply their own ideas of how the candidate is exploring the space and adequately testing it.

      Any interviewer can ask the candidate how they would write bugs or test cases against the Test Quest application and, in that sense, that would be no different than the subjectivity inherent between interviewers in asking the more generic questions of “How do you write a bug report” or “How do you write a test case”. What’s interesting in the game context is that since it’s so visual, I like to see how testers would make it very clear to a developer what they did or didn’t do in terms of, say, reproducing a bug.

  2. Tatiana says:

    NEAT!

    This is really cool. I did spot the bug in the screenshot!

    Out of curiosity, do you find you can give this challenge to only senior people? Is it fair to juniors as well?

    The fact that you can type commands to the game, which should expand the the number of tests. This reminds me of the ‘old-school’ text adventures but with a graphic overlay.

    • Jeff Nyman says:

      Out of curiosity, do you find you can give this challenge to only senior people? Is it fair to juniors as well?

      This could go to any candidate: junior, intermediate, senior. The level to which you would judge the candidate in terms of how they approach the challenge will differ based on experience level, of course.

      The fact that you can type commands to the game, which should expand the the number of tests.

      Correct. The game is meant to work at a few abstraction levels. There is a menu of controls that the player can access, which control the game in terms of how it operates (as you can see in the fourth screenshot). There is the ability to move around the game space (via mouse or keyboard). And then there’s the ability to type commands to the game (seen in the third screenshot).

      It’s interesting to see where candidates spend a lot of their testing time.

  3. Ryan Sofranko says:

    At first when I started looking at this, I thought you were somehow doing a variation of the “logic puzzle” tests. Remember those? Someone asks you why manhole covers are round or how many piano tuners there are in the United States. So I’m thinking “Okay, this guy just wrapped up puzzles in a more clever context.” But I looked at the supporting documents you provided and I can see what you mean about this not being a challenge to solve puzzles but rather a chance to explore and discuss. Color me impressed.

    Curious: do you find that people who do a lot of game playing as a hobby tend to do better at this than other candidates? Does that in any way have to be accounted for?

    • Jeff Nyman says:

      Perhaps interestingly enough, I have found that, in many cases, people who fashion themselves as “gamers” tend to do a little worse on this than those who do not.

      I haven’t isolated exactly why that is yet. Certainly one element of that is simply familiarity with the limitations of the medium. Gamers tend to know how buggy games can be and will often tacitly accept certain things.

      But even given that, that would not necessarily account for the fact that many people who tout themselves as “gamers” tend to do the most minimum of exploration before rendering decisions as to whether the game is ready to release.

  4. Lee says:

    I agree with the other comments.  This is brilliant.  I did not see the bug at first, but after going through the post, reading through the manual and looking back at the screenshots, the error prominently stood out.

    Is there any way to try Test Quest?  I am not sure if others have asked, but I would be very interested in running through the game and see what happens.  This is a fabulous way to interview, but I could see it also being used by someone as a way to evaluate their existing abilities and find the areas that need improvement.

    • Jeff Nyman says:

      Thanks for the comment, Lee. Yeah, absolutely I can bundle up a version of it and get it out to you. It contains an executable so what I’ll probably do is upload the bundle and, if you don’t mind an email intrusion, I’ll send you a quick email with the link for that.

  5. Roman says:

    Hi Jeff. Is there a way to get a copy of the game? It looks great and I would gladly use it for educational purposes as well as for self-evaluation. Thanks in advance.

Leave a Reply

Your email address will not be published. Required fields are marked *