Interview “Technical” Testers for Broad Skills

I recently did an interview for a “technical testing” position and it was one where they had you do a whole lot of exercises for coding. In fact, as far as I could tell, that’s pretty much what all the interviews were going to be. The interview was thus, to me, largely a waste of time. I actually told the interviewers that and, to be fair to them, they were quite gracious, did engage me in discussion, and actually seemed to agree with me on a few points. But here I want to talk about why I felt the interview was a waste of time and what role “coding exercises” have when hiring so-called “technical testers.”

The Interview Begins

Literally the first interview question started off with the interviewer saying:

“Show me how to reverse a string in any language of your choice.”

Okay. So I wrote this:

I got a blank stare for a bit and then the interviewer said:

“Yeah, but how would you do it?”

To which I responded with:

That is how I would do it.”

Another blank stare for a bit, and I could tell the interviewer was probably wondering what sort of idiot he was dealing with here. He finally said:

“Okay, but, tell me how you would do it without using a library.”

Ah. I knew where this was going to go but I said:

“Well, I didn’t use a library, really. ‘reverse’ is a built in method on String objects in Ruby.”

Interviewer’s response:

“Yeah, well, don’t use that. Just write your own.”

Yep. That’s exactly where I thought this was going. So I said: “Okay, so if I was doing it inefficiently and not reusing what is already there, I would do this …” and stumbled through an implementation of a string reverse function that looked something like this:

There’s probably better ways and I know there are more idiomatic Ruby ways. Here’s the thing: I don’t really care if there are better or more idiomatic ways or not because I wouldn’t go through this process unless the language I was using didn’t have the built in functionality and a known library didn’t already provide it. And if I did have to build my own, then rather than test my memory and see how well I could do it, I would probably just look it up online and see how someone else did it since, likely, someone already has.

I felt that exercise was a waste of time and I actually did let the interviewer know that. They sympathized but said they needed to make sure they were getting “technical testers.”

I said (roughly):

“I get it. But all you really proved is that I could code something that already exists. I get that it shows a bit of technical skill on my part, but it’s still not showing you the important things I can do. For example, nowhere in Lucid, Fluent, Symbiont or my other tools do I reverse strings. What I do is …”

And then I launched into that.

So What Am I Complaining About Here?

You know what would have been a much better question for them to have asked me, given the role they were filling? Something like this:

“Show me how you would start writing an automated test framework.”

Or maybe even:

“Write me any bit of code you want and explain to me what you are writing.”

I use that latter one myself when I’m interviewing candidates that have coding background and I’ve found it very instructive because what I really want to see is how they internalize what they are doing — whatever it is — and how they then make that internalization understandable to me. Do note that how I frame the question, and my expectations for a more open-ended question, is different in kind from the “show me how to reverse a string” and other such similar questions.

The Interview … such as it was … Continued

Anyway, during my interview here was another one that was thrown my way (by a different interviewer):

“Let’s say you have a boolean two dimensional array and each row is sorted. Write some code that will find the row with the maximum number of 1 values.”

Once again, I get it: these kinds of problems are good for considering optimizations. But, again, I saw this as a waste of time. And I told the interviewer exactly that — without, I might add, even trying for a solution.

Now, I know, I know — I can hear what you’re thinking from here: “Wow, what a jerk. Maybe they just wanted to see you try.”

Perhaps. But there were much better things they could have had me try, in my opinion, and those things would have told them much more about my skills not just as a test solution developer but as someone who applies test thinking to whatever I develop. Also, to be clear, I didn’t just not do the exercise. Specifically, I said something along the lines of:

A ‘technical tester’ might have to do something like that. Maybe. But I doubt it. What I don’t doubt is that using juggling algorithms to figure out array rotations may be something you have to do often in enterprise application development. What I do know is that it’s nothing I’ve had to do in writing test solutions.”

And this is where there is admittedly a gradation of skill and thus did I tune out the interview and start thinking about the interview, if that makes sense.

Reflecting on the Interview

As I continued engaging with the interviewers, for example, I told them that I’ve met many people who can do the problems they were giving in just a few seconds. Thus their ability to do so really meant very little to me. Those same people, however, could often not substantively discuss the following with me:

  • What’s the difference between a test framework and a test driver?
  • What are some common strategies or patterns for following a single-responsibility principle in modern automated test frameworks?
  • What are some common “test smells” that you’ve learned to look for when writing automated code?
  • How do you determine when to push a test “down the stack” (say, from system, to integration, to unit)?
  • What are the benefits of a polyglot approach to test solution development?
  • What are the dangers and benefits of having an abstraction layer in front of automated test code?
  • Are there certain design patterns that work just as well in automated test code as they do in enterprise level code?
  • Give me a rough idea of what a test framework based on convention over configuration looks like.
  • What are different strategies for handling test data in automated test frameworks?

I don’t care how good a coder is or what problems from computer science they can solve: someone who cannot answer the above questions for me will not do me much good as a test solution developer who can. And a “test solution developer” is, to me, one definition of a “technical tester”. That being said, depending on what I’m looking for, the inability to answer the above questions at all would tell me one thing. But the ability to converse to any degree at all about some of them would tell me another. To wit, no matter what answers I get, I learn how the person thinks, what they’ve been exposed to, how they’ve internalized what they’ve been exposed to, what they consider important, what they don’t consider important, etc.

I’m not saying coding exercises don’t have their place. They most certainly do, particularly if you are developer with a pure focus on development. When you are a test solution developer, I would argue those exercises still have their place — but — your “technical tester” ideally is one that has specialized by generalizing and thus there is more nuance.

Do you buy that? Well, I’ll state outright here that I’m basing that solely on myself and thus not only is it an entirely subjective point, but it’s a terribly conceited one as well. And, to be fair, maybe I’m just rationalizing for my own lack of deep skill.

But here’s the thing: I’ve seen plenty of “technical testers” that can write better code than me, can optimize that code, and so on. However where I can often surpass them is in focusing beyond the code and considering how all of those constructs relate to actual testing and how principles and idioms of effective test design must be encoded in those tool solutions. What I find lacking are “technical testers” who can tell the difference between a humanizing interface and a fluent interface, or an external API versus an internal API — or why any of that even makes a difference.

So, yeah, speaking for myself: I can code — to varying degrees — in languages like Clojure, Scala, Groovy, Java, C#, Python, Ruby, C++, and JavaScript. I am an expert in none of them — nor will I likely ever be. I’m not even what I would call proficient in all of them. In most cases, I’ve forgotten more than I’ve learned, in which case Stack Overflow is often my friend.

What I can do, however, is tell you why I would choose one language over another for a given test solution. What I can tell you is why I might combine certain solutions in different languages to get a polyglot effect. What I can tell you is what kind of testing ecosystem exists around those various languages. I can tell you why I might use JBehave over Cucumber-JVM and why I would never use Spinach but I might use Turnip. I can tell you why I might want to adhere to the Capybara API although why I think Watir-WebDriver actually makes a stronger case. I can tell you why I might opt for Mechanize rather than relying on the seemingly more robust Faraday. I can tell you the differences between Spock, GSpec, easyb, ScalaTest, Concordion, and so on. And, given enough operating context and some time, I could write up solutions to prove my points.

Can I code in everything I just said with equal efficiency and effectiveness? Not really. Could I do it ‘on demand’ in an interview? In most cases, not a chance. But I can do it. And I have open source projects that I work on which show that, for better or worse, I can throw code together that does work and can add demonstrable value.

Was There a Point?

Probably but I haven’t found the correct way to express it yet. I’m convinced that how companies look for modern, technical testers is wrong. I feel it’s because very specific skills are sought rather than more broad skills. Companies tend to look at “developers who want to test” rather than “testers who can do some development.”

So when interview challenges come up, and particularly when they involve coding, the question should always be how much and to what extent does coding prowess have to be proven in interviews for “technical testers”? If a tester has written their own frameworks or libraries but have only done so in the context of one language — and a limited subset of the language at that — it argues for a more nuanced approach to interviewing them.

The means of deciding what the “technical tester” means — what I would call the test solution developer — is an interesting question and I find various companies take very different approaches on this, not only based on what they consider “technical” but also on how broadly they consider “testing” in terms of it not just as a skill set but as a mind set. I’m much more concerned with the latter since I can much more easily train on the former.


This article was written by Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

7 thoughts on “Interview “Technical” Testers for Broad Skills”

  1. Good read…great post!

    < Now, I know, I know — I can hear what you’re thinking from here: “Wow, what a jerk. >

    Close. I replaced “what a jerk” with “my hero!” J

    < I’m convinced that how companies look for modern, technical testers is wrong. I feel it’s because very specific skills are sought rather than more broad skills. >

    As I was reading your post, I was taking notes. Before I got to the sentence above, when reading about your affinity for Stack Overflow, I wrote, “I value general problem solving skills more than specific language knowledge. Know how to solve many problems, not few problems.” Then, I got to the sentence above, and saw that you’d already reached that conclusion yourself.

  2. I agree with your general sentiment but the two dimensional array problem?  It’s perfectly reasonable to ask a question that demonstrates that you at least have basic coding skills.  Sure, this could be used to determine if you understand how to optimize code but just being able to solve proves that you can write code.  People lie and exaggerate on their resumes a lot.  How do they know that you can even pass the Fizz Buzz test?

    Your points about other more involved questions are valid, but I personally have ruled out candidates because they could not pass a simple coding test in ANY language they claim to know.

    1. Matthew, thanks for the comment! Much appreciated.

      The problem with the specific two-dimensional array problem — for me, anyway — when applied to a test solution developer is this: I only ask questions where, for each question, I could say that I would reasonably not consider the candidate if they didn’t do well on it.

      This array problem/challenge would not fit that criteria because for a test solution developer — as opposed to an enterprise developer — this situation has so little resemblance to what you’ll be doing day-to-day, that it won’t be an operative question for me. Could it tell me something? Perhaps. But not what I want to know. I work with many test coders who probably couldn’t answer that challenge and yet they are very good coders in their arena. (I consider myself one of them.)

      You can test if people can write code; but test for whether people can write the relevant code for the type of domain you are in, I guess is my main point. So if a test solution developer has, say, Selenium or Capybara or whatever else on their resume, I’m going to ask them to show me code in that arena. If they have written their own test solutions, and promoted them on their resume, as I do, then I’m going to ask them to code something utilizing that solution and prove to me that it works.

      This is really focused on the “technical” tester for me and thus on the test solution developer. I think too many companies treat those interviews as if they were interviewing a developer first and foremost — and that is what I think is misguided.

      For example, like yourself, I’ve had issues with people who clearly can’t code at all. But that’s not interesting to me since it’s easy to ferret out; sometimes even in the phone screen. What I’ve found that is generally easy for “developers” of any stripe is simply being given a problem to solve. (“Here, candidate, solve this array problem.”) That (often, but not always) tells me little. However, I’ve found those same coders often stumble when I give them some freedom: “Candidate, describe what you’re going to code for me, code it (in your language of choice), and then tell me how it works.” All the sudden those great coders, who can solve array optimization problems in their sleep, start to show me what I really want to know: How do you pick from a reasonable problem space to solve and demonstrate? Do you code what you said you would code? Did what you code do what it was supposed to? Are you able to explain it after the fact?

  3. I hear what you’re saying but I feel like a more relevant coding question is not going to be one which you can easily squeeze into an interview in real time.  For example, at my current company, we expect all testers to be able to program in Java.  Part of our interview process involves a take home testing/coding assignment.  While not perfect, it does hit a sweet spot in measuring a candidates’ coding and testing skills/intuition.

    String/data structure/algorithm interview questions don’t really prove that a technical tester is a good tester, but they demonstrate a basic level of programming skill (maybe an advanced level for the trickier questions).  Also, I’m sometimes in the position where someone else has phone screened a candidate and I’m double checking that the person has technical skills. . . I’m QA’ing the phone screen if you will. . .

    Asking someone in real time how to code a Selenium solution is a bit unnatural IMO. . . you never implement a solution on the fly like that.  Or rather, if you ask this type of question, do you give them access to a computer with all the typical resources that you would have at hand: google, StackOverflow, an editor, etc.  As artificial as an abstract programming problem may be, it lends itself better to a white board solution than something that involves Selenium/JUnit/whatever.

    Anyway, I just discovered your blog recently, great content.

    On a related note, I ran across this online QA screening quiz for Yodle:  I like it, it seems like a decent weeder for real-world-ish technical problems.


    1. Good points, Matt.

      Consider the “Bowling Game” example which is very common now in Ruby shops. There you have a candidate write a feature file (Cucumber) that tests an existing bowling score implementation (that you provide them) and then they have to write tests (in whatever framework they choose) to cover the implementation. Along the way, as you might guess, the implementation has some bugs.

      That interview can easily be done in one hour. Granted, it does depend on the skill level of the candidate. The goal isn’t that they “finish” so much as they simply work through it. A similar solution is done in Java, just using Cucumber-JVM (or JBehave) and JUnit (or TestNG).

      I agree with you about the possibility of white boarding concepts if it’s something that might be hard to implement. For example, a lot of candidates I talk to have QTP experience. Well, my company isn’t going to buy me a QTP license just to interview. However, I’ve used QTP in the past so what I do is provide the candidate with working QTP code (using its descriptive programming model) and then I provide them another piece of very streamlined code (also in QTP). The latter is their target. I ask them to tell me how they would reconstruct the provided code into the target. (This requires them to understand classes in QTP among other things.)

      Considering the Selenium example, however, if someone has indicated good skills in that, there’s nothing too hard about having them show you how they would automate either a fake site that you create or a real one. (I created a fake web app called Dialogic that I can use for this.) A company called SpiceWorks actually has you automate a section of their site during the interview process, and you can use Selenium, Capybara, Watij, or Watir. A company called SpaceX has you go through a Python-based test implementation.

      The good thing for me about all these is that they can show me different levels of skill. For example, with Selenium and related tools, some candidates immediately start creating a page/activity object pattern. Others write little helper methods and call out to those while others just cram everything into one big method that performs the test. Some like to create a factory method. Some use a data builder pattern. Some use a fluent interface and some do not. And even if they don’t finish their implementation, I’ve seen enough to know that they understand (or do not understand!) not just some coding, but some good practices around the architecting of that code.

      Along with this, of course, a technical tester — whether SET, SDET, or Test Solution Developer — has to also be able to think like a business analyst. That means they need to be able to recognize a good test or a bad test. And that’s the case whether that test is in code or in writing (or sometimes in both).

      All this said, you are correct: there is often never as much time as you’d like to interview the candidate in the way that you’d want. So you have to get creative. For example, I wrote a video game called “Test Quest” — styled after the old-school graphical adventure games. I ask the candidate to play it with. We “pair test” as it were. They are given a set of requirements for the game. They are given the solution to the game (so they don’t have to be a gamer). The source code is available if they want to look at it. (I purposely did it in a Lisp-like language so not too many candidates have an advantage.) This doesn’t teach much if anything about coding since the exercise is focused on a different set of thinking skills, but I bring it up just because I believe it’s important to be as creative as possible in coming up with fair challenges that really allow you to dig deep into mind set first, skill set second. (The latter can be trained on easily; the former not so much.)

      Great discussion. I really appreciate your comments and thoughts.

  4. This is interesting, Jeff.

    I agree that most “coding challenges” that have a focus on hiring technical testers is very limited and un-nuanced right now.

    I do know some companies like to give the “do it at home” challenge, where they basically give you a problem, usually coding-based, and ask you to work on it and then send it back. Sometimes that can be useful but those get perilously close to “work for free.” Technically it is part of the interview process, but those challenges are sit wrong with me.

    1. Ironically, while I can see your “working for free” viewpoint, I would actually argue most candidates should prefer this mode because it takes some pressure off. After all, you can stumble your way to a solution regardless of how inefficient you are at at!

      And I guess that highlights my problem with the “homework” challenges is that you miss out on a critical thing: watching the person as they solve a problem. You miss out engaging with them as they discuss what they are doing. These kind of challenges are typical developer thinking which removes much of the human element and instead focuses on the coding challenge. I realize why companies do these kinds of exercises and it can certainly make sense, particularly given logistical realities. But the fact is that the interaction with a person as they engage with a problem is critical.

      If you read my comments to Matt, I mentioned the test game I provide. The whole value in that is my ability to sit with the person, as they explore the game and decide how to test it. I need to see when they get frustrated. I need to see the paths they try and don’t try. I need to see how they maximize their time.

      Likewise with pure coding challenges: there are so many examples out there that you can crib from BitBucket or GitHub that even if the candidate provides a great example, I would still then want to go through it with them anyway. So I may as well do that with them in person.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.