I recently did an interview for a “technical testing” position and it was one where they had you do a whole lot of exercises for coding. In fact, as far as I could tell, that’s pretty much what all the interviews were going to be. The interview was thus, to me, largely a waste of time. I actually told the interviewers that and, to be fair to them, they were quite gracious, did engage me in discussion, and actually seemed to agree with me on a few points. But here I want to talk about why I felt the interview was a waste of time and what role “coding exercises” have when hiring so-called “technical testers.”
The Interview Begins
Literally the first interview question started off with the interviewer saying:
“Show me how to reverse a string in any language of your choice.”
Okay. So I wrote this:
I got a blank stare for a bit and then the interviewer said:
“Yeah, but how would you do it?”
To which I responded with:
“That is how I would do it.”
Another blank stare for a bit, and I could tell the interviewer was probably wondering what sort of idiot he was dealing with here. He finally said:
“Okay, but, tell me how you would do it without using a library.”
Ah. I knew where this was going to go but I said:
“Well, I didn’t use a library, really. ‘reverse’ is a built in method on String objects in Ruby.”
“Yeah, well, don’t use that. Just write your own.”
Yep. That’s exactly where I thought this was going. So I said: “Okay, so if I was doing it inefficiently and not reusing what is already there, I would do this …” and stumbled through an implementation of a string reverse function that looked something like this:
chars = string.length
reversed_word = ''
while chars > 0
chars -= 1
reversed_word += string[chars]
There’s probably better ways and I know there are more idiomatic Ruby ways. Here’s the thing: I don’t really care if there are better or more idiomatic ways or not because I wouldn’t go through this process unless the language I was using didn’t have the built in functionality and a known library didn’t already provide it. And if I did have to build my own, then rather than test my memory and see how well I could do it, I would probably just look it up online and see how someone else did it since, likely, someone already has.
I felt that exercise was a waste of time and I actually did let the interviewer know that. They sympathized but said they needed to make sure they were getting “technical testers.”
I said (roughly):
“I get it. But all you really proved is that I could code something that already exists. I get that it shows a bit of technical skill on my part, but it’s still not showing you the important things I can do. For example, nowhere in Lucid, Fluent, Symbiont or my other tools do I reverse strings. What I do is …”
And then I launched into that.
So What Am I Complaining About Here?
You know what would have been a much better question for them to have asked me, given the role they were filling? Something like this:
“Show me how you would start writing an automated test framework.”
Or maybe even:
“Write me any bit of code you want and explain to me what you are writing.”
I use that latter one myself when I’m interviewing candidates that have coding background and I’ve found it very instructive because what I really want to see is how they internalize what they are doing — whatever it is — and how they then make that internalization understandable to me. Do note that how I frame the question, and my expectations for a more open-ended question, is different in kind from the “show me how to reverse a string” and other such similar questions.
The Interview … such as it was … Continued
Anyway, during my interview here was another one that was thrown my way (by a different interviewer):
“Let’s say you have a boolean two dimensional array and each row is sorted. Write some code that will find the row with the maximum number of 1 values.”
Once again, I get it: these kinds of problems are good for considering optimizations. But, again, I saw this as a waste of time. And I told the interviewer exactly that — without, I might add, even trying for a solution.
Now, I know, I know — I can hear what you’re thinking from here: “Wow, what a jerk. Maybe they just wanted to see you try.”
Perhaps. But there were much better things they could have had me try, in my opinion, and those things would have told them much more about my skills not just as a test solution developer but as someone who applies test thinking to whatever I develop. Also, to be clear, I didn’t just not do the exercise. Specifically, I said something along the lines of:
A ‘technical tester’ might have to do something like that. Maybe. But I doubt it. What I don’t doubt is that using juggling algorithms to figure out array rotations may be something you have to do often in enterprise application development. What I do know is that it’s nothing I’ve had to do in writing test solutions.”
And this is where there is admittedly a gradation of skill and thus did I tune out the interview and start thinking about the interview, if that makes sense.
Reflecting on the Interview
As I continued engaging with the interviewers, for example, I told them that I’ve met many people who can do the problems they were giving in just a few seconds. Thus their ability to do so really meant very little to me. Those same people, however, could often not substantively discuss the following with me:
- What’s the difference between a test framework and a test driver?
- What are some common strategies or patterns for following a single-responsibility principle in modern automated test frameworks?
- What are some common “test smells” that you’ve learned to look for when writing automated code?
- How do you determine when to push a test “down the stack” (say, from system, to integration, to unit)?
- What are the benefits of a polyglot approach to test solution development?
- What are the dangers and benefits of having an abstraction layer in front of automated test code?
- Are there certain design patterns that work just as well in automated test code as they do in enterprise level code?
- Give me a rough idea of what a test framework based on convention over configuration looks like.
- What are different strategies for handling test data in automated test frameworks?
I don’t care how good a coder is or what problems from computer science they can solve: someone who cannot answer the above questions for me will not do me much good as a test solution developer who can. And a “test solution developer” is, to me, one definition of a “technical tester”. That being said, depending on what I’m looking for, the inability to answer the above questions at all would tell me one thing. But the ability to converse to any degree at all about some of them would tell me another. To wit, no matter what answers I get, I learn how the person thinks, what they’ve been exposed to, how they’ve internalized what they’ve been exposed to, what they consider important, what they don’t consider important, etc.
I’m not saying coding exercises don’t have their place. They most certainly do, particularly if you are developer with a pure focus on development. When you are a test solution developer, I would argue those exercises still have their place — but — your “technical tester” ideally is one that has specialized by generalizing and thus there is more nuance.
Do you buy that? Well, I’ll state outright here that I’m basing that solely on myself and thus not only is it an entirely subjective point, but it’s a terribly conceited one as well. And, to be fair, maybe I’m just rationalizing for my own lack of deep skill.
But here’s the thing: I’ve seen plenty of “technical testers” that can write better code than me, can optimize that code, and so on. However where I can often surpass them is in focusing beyond the code and considering how all of those constructs relate to actual testing and how principles and idioms of effective test design must be encoded in those tool solutions. What I find lacking are “technical testers” who can tell the difference between a humanizing interface and a fluent interface, or an external API versus an internal API — or why any of that even makes a difference.
What I can do, however, is tell you why I would choose one language over another for a given test solution. What I can tell you is why I might combine certain solutions in different languages to get a polyglot effect. What I can tell you is what kind of testing ecosystem exists around those various languages. I can tell you why I might use JBehave over Cucumber-JVM and why I would never use Spinach but I might use Turnip. I can tell you why I might want to adhere to the Capybara API although why I think Watir-WebDriver actually makes a stronger case. I can tell you why I might opt for Mechanize rather than relying on the seemingly more robust Faraday. I can tell you the differences between Spock, GSpec, easyb, ScalaTest, Concordion, and so on. And, given enough operating context and some time, I could write up solutions to prove my points.
Can I code in everything I just said with equal efficiency and effectiveness? Not really. Could I do it ‘on demand’ in an interview? In most cases, not a chance. But I can do it. And I have open source projects that I work on which show that, for better or worse, I can throw code together that does work and can add demonstrable value.
Was There a Point?
Probably but I haven’t found the correct way to express it yet. I’m convinced that how companies look for modern, technical testers is wrong. I feel it’s because very specific skills are sought rather than more broad skills. Companies tend to look at “developers who want to test” rather than “testers who can do some development.”
So when interview challenges come up, and particularly when they involve coding, the question should always be how much and to what extent does coding prowess have to be proven in interviews for “technical testers”? If a tester has written their own frameworks or libraries but have only done so in the context of one language — and a limited subset of the language at that — it argues for a more nuanced approach to interviewing them.
The means of deciding what the “technical tester” means — what I would call the test solution developer — is an interesting question and I find various companies take very different approaches on this, not only based on what they consider “technical” but also on how broadly they consider “testing” in terms of it not just as a skill set but as a mind set. I’m much more concerned with the latter since I can much more easily train on the former.