There is still so much wrong with how testers — even those who will write automation — are interviewed. I talked about this already regarding how technical test interviews are broken and about interviewing technical testers more broadly. Is there really more to say? I think so; let’s see if you agree.

Lately I’ve having and hearing some interesting discussions around things like code-based challenges, such as Hacker Rank, for interviews. And in many strict developer contexts — i.e., where programmatic skills are your speciality — those seem to make sense. But here I’m speaking about testers. Testers who may have to have some technical acumen; they may have to program; they may even have to act like a developer to a certain extent in some situations.

In fact, I think testers must possess technical acumen. They should have the foundations of programmatic skills. They do act like a developer in not just some, but most situations.

But code-challenge interviews that treat technical testers like specialist developers are flawed. Actually, the title of this post notwithstanding, I’m actually not talking about interviewing a tester as a developer, but rather the situation where testers are interviewed as programmers. The industry conflates the role of developer and programmer sometimes. I put that little ambiguity in on purpose and it will permeate this post.

For the purposes of this post, I’ll stick to my own past experiences. As I go through all this, think about the distinction between programmer and developer. Also think about the distinction of a “technical tester” and what kinds of programmatic skill would most demonstrate their speciality as a tester.

Utilize What People Present

When I send my resume in to a company, it has, very prominently, my GitHub repo link. That GitHub repo, if someone bothered to look, has a series of open source test solutions that I’ve written as well as some fairly comprehensive side projects. I have stuff in JavaScript, Ruby, Python, and Java.

I can tell you that when I get a resume with a GitHub (or BitBucket, whatever) repository, I go check it out. This is particularly true if the person being interviewed is a tester. But then I also use what I discovered. I very specifically ask them about it. Some questions I’ll ask:

• Which project are you most proud of that you worked on?
• What was one of the hardest things you had to program on one of those projects?
• Walk me through the code for one of your projects.

I learn a lot from this kind of discovery, including their demonstrated passion (or lack thereof) for some projects they clearly spent time on. Granted, not every person you interview will have this. But when they do, use it!

So you can imagine my disappointment in my past where I was rarely asked questions like these. And, interestingly, I was almost never asked them when the interview component was actually largely technical. I’ve been asked more about my GitHub content by non-technical interviewers than technical ones.

And that is very broken.

I’m going to generalize a bit here but I’m basing this on my experience: I’ve more often than not eventually found the reason for this is because those technical interviewers don’t have the skills to judge something beyond the algorithm exercises they give you. Or, rather, they do. The interview process is simply not flexible enough or imaginative enough to adjust for a possibly more interesting discovery experience.

But let me give you a few examples of some “standard” challenges and how I think they could be framed when your goal is looking for a technical tester who can reason about problems as well as look at and reason about code, rather than showcase computer science.

Palindromes

Palindromes are a challenge that some people like to give. Here’s an example that I cobbled together of doing that in Ruby:

The fact that I could eventually figure out how to write that is less interesting — or should be — than whether I can start coming up with ways to test it, such as:

palindrome?("Madam, in Eden, I'm Adam")
palindrome?("A man, a plan, a canal: PANAMA!")
palindrome?("A1B2B1A")
palindrome?("A1B2B2A")
palindrome?("12321")

By the way, a lot of code-based challenges would not let me use something like phrase.reverse above. They prefer you to not use existing library features but, instead, write out the entire algorithm. That might tell you something about a specialist developer and how they can program. That does not tell you nearly as much about a specialist tester and how they can test or even write test solutions.

Find Sum of Two Numbers

Another very common example is something along the lines of: given an array of integers, find two numbers such that they add up to a specific target number. Here was an example I found out how to do in Ruby:

The key phrase there is “found out how to do.” I started looking things up and I found that interesting example with combination(2).find_all and I had to learn what that actually meant. I knew it worked. But I wasn’t sure why. And I realized that my attempt to simply Google a solution, check if it worked, and then figure out — and explain — why it worked was so much more crucial than whether or not I could write it from memory or based on my algorithmic knowledge.

Most times, however, you aren’t given that chance. You are told to stand up at a whiteboard and write it out from scratch. Even though, in reality, you would in fact look it up if you didn’t know it, try out an implementation you found, read up a bit on whether that implementation is any good, and then test it a bit to see if it works as expected.

So why don’t many technical interviews get framed that way? And, again, most of this had little to no bearing on my ability to test and not even to write test solutions. Ostensibly what I was being hired for at the time.

Character Frequency

Speaking of that, here’s another one where I did something similar to the above. The idea was to write a program that, given a string, outputs a new string that contains each character in the original string exactly one time, followed by the frequency of occurrences of that character in the original string, and in alphabetical order. Here’s something I found that does it:

I didn’t just find that exact bit of code. I had to do a few searches on StackOverflow to some generic questions about “how to do x” and then I put them together into the above. That process, including me being able to explain what I ended up with, is much more valuable than seeing if I can concoct that — or any other solution — from memory or at a whiteboard.

PITCH Data Example

Here’s an example you might get in trading or stock type environments. You are basically told:

Write a program which reads PITCH data from standard input and, at the end of the input, shows a table of the top ten symbols by executed volume. For example, your table should look something like this …

And then you are given something like this:

SPY  24486275
QQQQ 15996041
XLF  10947444
IWM  9362518
MSFT 8499146
DUG  8220682
C    6756932
F    6679883
EDS  6673983
QID  6526201

You are also provided with a PITCH specification that is slightly over twenty pages in length. Basically what you need to do is read “Add Order” messages and remember what orders are open so you can apply “Order Cancel” and “Order Executed” messages. It gets complicated because “Trade Messages” are sent for orders which were hidden. So that means you need to use both Order Executed and Trade Messages to compute the total volume.

I can’t post everything about this project but I did make available a few bits of it on GitHub.

This exercise does very little to show how a tester should be thinking about these things. This is very much a developer-focused exercise. From a test perspective, it’s important to understand how someone conceptualizes testing, how they would treat data, whether they consider incidentals as part of test data, the amount of tests they would apply to a given situation (too many? too few?), and so on.

There was no expected output provided with this example. We had what the output should look like. But not what the outcome of the operations should be. And that’s important. Testers care about outcomes, not just outputs. So, as such, it was not possible for someone to determine if their implementation worked. You could tell to a certain extent if it wasn’t working, but you could never be sure it fully worked. So these expectations should be provided. When testing applications we generally do have an oracle of what we expect to find in a given situation. If we do not, part of what testers do is ask for one.

I explained all this to the hiring team after I wrote about a hundred lines of Python code and created three data sets to test out some of my assumptions. I was actually pretty proud of what I came up with even though I knew it wasn’t “finished.” The interviewers clearly could not have cared less about any of what I was providing. Which is exactly what I would have been providing had I worked there, including challenging the assumptions.

Sudoku Example

Sometimes you are given a Sudoku challenge, which is basically algorithmic in terms of how you recognize that you have a solution to a Sudoku board. I recently put up an example of a possible implementation that I worked on awhile back.

This was actually a fun challenge, to be honest. And I had to do two implementations because I couldn’t figure out how to do the reporting one in the context of my original solution, which I do indicate in the README. The problem is I spent more time programming this then actually coming up with different test solutions for it. And, keep in mind, I’m interviewing as a tester.

Better would have been to provide the implementation and then ask me to come up with test and data conditions. For example, you can see some of the boards I came up with.

But Can You Test? Can You Write Test Solutions?

Now, again, please keep in mind the full context here. I do believe testers have to be “technical.” By that, I do believe they have to understand programming basics enough to create test solutions or utilize existing ones. But I believe, first and foremost, they have to test. They have to think like a tester. They have to think around implementations and find the sensitivities in those implementations.

So considering a test candidate that you want to have some developer (not just programmatic) skill, consider the following:

• Provided with an implementation of something, how would they test it? How do they frame input and output conditions? Do they seem to include incidentals in the test?

• When they have to use an implementation of something, such as a test library, framework, runner, etc, it’s important to see how they make those choices. Why did they go with one approach over another? Why would they choose a particular library over some other one?

• When they have to provide an implementation of something, such as a test solution, it’s much more important to understand how they craft those solutions. For example, where does test data live? Do they use an existing runner and, if so, which? And why? Do they try to wrap APIs, like Selenium or Appium, around other APIs? How? And, more importantly, why? Do they use specific patterns (context factory, data builder, proxy, etc)? If they are testing APIs or services, how do they break up the endpoint in a test? Do they have separate methods for the host and the parameters? How much of the response do they assert?

Developer who Tests? Tester who Develops?

If you are someone who conducts technical interviews with a code-challenge component and you are interviewing testers, it is important to know which category you are placing that person into: a developer with some testing knowledge or a tester with some development knowledge? There are a lot of key differences in that distinction.

And, once again, I’ll note that what we’re really talking about here is not a developer, but a programmer. As a hint for an upcoming post, I do believe that testers are developers. They certainly work in a development context and do have to possess many of the thinking skills of a developer. And some of that does require specialist programming skills. In this case, importantly, that speciality is around crafting test solutions.

So in that interview context I just mentioned, check for that kind of specialist programming.

But it won’t hurt to check if the person can actually test as well.