Here I’m not speaking to the people who are interviewing for roles in automation. I’m speaking to the people hiring them. The interview process is entirely broken in so many places. According to Eric Elliot, code-based interviews have always been broken. And he’s probably right. Sahat Yalkabov said something similar. He’s probably right too. But here I’m focusing on the companies and hiring managers that are exacerbating the technocrat problem. So let’s talk about this.
Actually, this is like beating a dead horse so I’ll keep this brief. I’ve already talked broadly about this regarding interviewing technical testers for broad skills and regarding interviewing testers as if you want testers.
Companies are still doing things like FizzBuzz, Anagrams, summing up numbers in an array, etc. I saw one about building a Sudoku solver. Now I get it: knowing algorithms can be useful. But there’s actually a lot more that’s useful for automation.
So let’s say your primary role, in the context of testing, would be automation in a given role. Maybe someone is even going to call you a SDET or SET. All of those things I just mentioned are absolutely not things that a test solution developer in that context would tend to code. So why do companies still waste time with these for technical testers?
I would rather see something like what I present here: Amazon Automate. This is the basis for a Selenium challenge, essentially using Amazon to see if someone understands how to write tests and code them up. That works because you’ll probably use Selenium a whole lot.
So, hiring managers, if you are interviewing candidates for an automation-style role and (1) you are not making sure that person is a good tester first and (2) you are focusing on the oft-used “algorithms” checks — you’re doing it wrong!
Automation is not testing. Automation is a particular technique; it is one that can be applied in the context of testing. As such, automation is not about writing a solution for the “FizzBuzz” problem. Automation is not about seeing if you can write code that will check if two digits in an array sum to a target value. Automation is not about seeing if you can solve palindromes or isograms.
Automation is about making decisions about technology such that it augments the testing work of humans, without the suggestion that it can replace it. Thus you want automation that is going to scale; that is going to be maintainable.
You want automation that is going to have a specific design philosophy for how it is constructed. You want automation code that is going to employ reasonable patterns in a way that is relevant to how test and data conditions are expressed and executed. Automation is often about choosing the right level of abstraction for consuming tests and executing them as checks, to use the current distinction of “testing” vs “checking.” (That is something I believe we testers get wrong.)
Given all this, if you are the kind of interviewer that I’m talking about, why are you interviewing for these roles as if you were interviewing for an enterprise developer?
Stop wasting everyone’s time, including your own.
Jeff,
Spot On! I would love to send this to some past companies I interviewed with and pulled these stupid tests on me. A couple of times I was about to look at the interviewer and just say what the hell does this have to do with Test Automation and my ability to do it. Very frustrating.
Only thing I can say is that it has made me rethink my skills and level of them for this line of work. The push for Selenium has caused a concomitant push on upping ones abilities to program with languages such as Java, C# and Python. But I don’t think I need to be a high-end developer in those languages to do the test automation work. I bring other skills and experience to the table regarding this type of work, and that is getting looked over. Which again is frustrating.
Add this one to your list of questions; solve a calculus problem (simple Integral). Yep, I got asked that one and I responded that I haven’t done that in 30 years since college. This caused this person to say I gave up too easily. What a fracking stupid question. And yes, this was for a Silicon Valley based company.
My favorite ones recently were writing a “reverse Polish notation parser.” (Ah, finally! Something that comes up all the time in testing!) I also had to write a “flood fill algorithm.”
I ended up just Googling both terms, found some code examples, worked to understand why they worked as they did and submitted them as-is. When asked about “my” solutions, I just basically regurgitated what I read and the brief bit of my own exploration.
And I totally passed! So what the interviews couldn’t do is distinguish someone who actually solved the problem and someone who just took someone else’s work. I’m sure they could have dug in and quite quickly figured that out. But the point is they didn’t. And most of these places don’t. They simply assume if you can do it — however you did it — it must be good enough. Which sort of defeats the purpose, I would presume, of checking actual understanding or ability.
While I have posted here about test automation interviews, I started the post with links from developers who feel the same way. I meet far more developers who have echoed these sentiments through the years than I do testers. There is the current “Glassdoor Underground”, for example. It’s made up of developers who post in the Interviews section exactly what code examples were asked for during the interviews. Usually solutions aren’t provided. Rather, it’s just to give everyone a chance to look it up beforehand.
Most developers I know are aware of the new “phoning it in” term. This is where you use those Glassdoor hints to search out the code examples, have them ready to go on your phone, and simply type them in or write them on the whiteboard.
When this is where things have gotten to — I think the process is broken.
Well, I think I could disagree more, but not much.
When interviewing testers, I don’t really care how familiar are they with Selenium – if they know how to code, I’ll teach them that in a week (probably less). I can’t ask them to write a complete framework, because that takes ages (heck, writing the first test in a framework is a long work to do), so any reasonably-sized “write a test for that” exercise I give will produce sub-standard code and won’t tell me much.
Instead, I would rather hire people who understand code and will be able to adapt to whatever is needed – when after 5 years of working on a product that was mainly web-based we started working on an API based application I could take 99% of what I know about test automation and use it (or at least have a meaningful discussion about it), so the excuse “This is what they’ll do” is a rather weak excuse to use for giving a tool-based question. Unless I’m hiring for something that I don’t already have, I don’t need the new hire to know Selenium, TestNG, Spring,Hibernate or even Java (all are tools we use).
Giving a simple “reverse this string in pseudo-code” gives me enough information about the coding skills of the candidate, without disqualifying people that don’t have that specific experience (or judging them by their ability to Google good examples).
If a candidate can describe what testing needs to be done on somehing, and can code, chances are that the they can code their tests properly with some guidance.
Personally, whenever I see a tool-based interview question, I cringe a bit.
I appreciate the counterpoint!
“Understand code” is too vague and way too broad, as we would probably agree. But it is important no doubt. What I look for is the ability to understand the relevant kind of coding that someone would have to do.
If you don’t care how familiar they are with Selenium — a tool they will presumably use a lot — that’s interesting to me. And I say that because you seem to suggest using Selenium is just using the API (“teach them that in a week”). How about using effective scrolling with Selenium? How about using a mutation observer so that you can be JavaScript-library agnostic (so you don’t have to worry about React, Angular, etc. differences)? How about using Selenium in the context of a data builder?
Using Selenium isn’t just about using its API. It’s being able to “adapt to whatever is needed” — as you said — when Selenium is run up against situations that crop-up all the time regarding waiting, scrolling, consuming test data, providing a data builder, and so on. It’s also about the types of abstractions that testers apply. Consider the shopping cart code for my Amazon exercise. Plenty there to show coding. In fact, very little Selenium. But there is some. So actually you get a bit a both: solving a problem and using Selenium. But the problem being solved is contextually relevant and one likely to be dealt with regularly.
Reading a script like this — assuming it was written by an automation candidate — tells me a whole lot about how that candidate frames automation coding. The use of a context factory is immediately obvious. Why take that approach? It can be useful — but it has its downsides. I’d like to hear from the candidate on that code choice? (If they just copied-and-pasted someone else’s example, they would likely not be able to answer that well.) There’s also some data there that I’m not sure where it’s coming from (‘item one’, ‘item three’) and I would want to know about that.
Consider the product page model. I love the use of dom_updated? checks (mutation observers, maybe?) — but still a sleep statement due to timing and display issues. But … isn’t that what dom updaters are supposed to deal with? I would want them to expand on their code comment.
When looking at their Amazon model I see — on line 40 — they do the same trick. But without the need for sleep. Why did the technique work in one area and not another? So what’s going on there?
I also notice (line 59) they do expectations in the model, not in the test. But I saw the use of assertions in the test script. Why is that? And where do they believe these things should go? The tests or the models? Clearly they do both — but was that conscious choice? Is the fact that assertions and expectations used part of how they frame their coding? If so, does that do something for me in a report.
Also with the script, I notice they wrapped the tool with something like RSpec. I would be curious about that. Choosing test runners matters and here they’ve shown me a choice. Was it just a default? Or do they have strong feelings about these types of English-style runners?
Just by this simple example I can learn a whole lot about how a tester conceives of tests, automation, expectations and assertions (which they apparently distinguish between), where to place the latter, what kind of techniques to use to traverse or iterate some elements, how they handle problems like waiting or display issues, etc. I learn their tool choices and their design decisions across a broad array of fronts.
Or I could just ask them to “reverse a string.” Admittedly, it would be less work for me to check.
This is so absolutely correct.
I’m one of the enterprise developers you mention and this process has long been broken even in the development hiring process, much less the automated testing process. I’ve personally seen technical testers who got hired because they could pass a code challenge but struggled with implementing good automation. It’s not that they couldn’t write code for the automation. It was that they didn’t think like a tester first, which you indicate.
Great article! I’m sharing this far and wide. We have an “emperor is wearing no clothes” situation and not enough people, tester or developer, are talking about it.
Jeff,
Great article. Your points made to Amit I think really drive home the message and would have made a great post/story in and of itself.
I think the fact that you personalize it. You tell how and why using the code samples helps you determine the coding thought process of a candidate. Sharing why you find a particular example interesting and why it helps you determine a candidate’s thought process.
When I interviewed candidates I only cared that they knew the how to code at a basic level. I spent a majority of the time trying to determine if they can think critically, speak clearly, express their ideas clearly, work well in the current work environment and most importantly solve problems.
(For some reason, none of my browsers will allow commenting on the same thread, sorry).
Thanks for the reply, it was quite interesting to read.
I must say that I don’t really get you when you say that “understand code” is vague – it might be a bit difficult to define in words, but like many other things – you know it when you see it. Sure, every system is a bit different, but understanding the basics is quite common – unless you code in assemly or optimize certain properties that are extremely important for your product, one can go quite a long way with understanding data structures, some basic memory allocation, concurrency and OOP.
As far as suggesting that using Selenium is just using the API – yes, that just about sums it up. All those cool features and ideas you mentioned? I DON’T CARE.
I don’t recall where did I hear that, but I heard that in selenium heavy projects, the selenium libraries are about ~20% of the code base. In my current job, 3-5% would be a far better guess, and I suspect that projects with 20% or more are not well engineered.
I don’t care about those cool ideas for several reasons:
Much of the stuff you mention are easily taught, and reinforced during code-reviews. If someone is choosing bad locators, or is not familiar with the best way to interact with our product when using web, I’ll be there to help and to comment about it during the code-review, or sooner.
I only care for optimizing what interrupts me. When I use sub-optimal scrolling strategies what do I lose? Execution time? Brittleness? I expect the coders we hire to address the problems they encounter – which might be selenium related, or might be related to something else – at any rate, googling for an answer is something we all do quite a lot. I can’t ask for knowledge in just about every tool or library we use, and selenium is not such a major library (There are some more notorious libraries doing way more serious magic than making a browser move).
I respect the time of the candidates I’m interviewing. Asking for 4-5 hours of their free time before the interview is quite a lot. Building even something like what you have in your “initial commit” is something I would put at 8-10 hours of work, for someone who is familiar with all of the tools you used (For me, as I never worked with Ruby, let’s put it at ~30 hours at least).
I don’t want to disqualify people who are used to a different tech stack. The code you linked to tells you quite a lot, but for me it is worthless – I don’t speak Ruby, and the only things I can say is that this code looks unintelligible for me, which might indicate a naming problem, or missing information about Ruby coding habits and style – a good Ruby coder will do just fine in our context, once we ramp them up with Java, but they would do horribly in any home-exercise, since there are some distinct differences in how you do stuff in each language.
I’m not asking them to build everything from scratch – Most of the time we’ll be working on an existing project, with an established framework – they will contribute and improve it, but there’s a reference to “how we do stuff here”, and if they don’t agree with it – we can have a meaningful discussion.
When I look at my coding tasks, There isn’t much time I spend with any of the libraries I’m using.
Sure, there was the time where I struggled for a week with TestNG to make it do what we wanted, and another in which I wrestled with Allure framework to get it to behave the way I want to, and the time I spent investigating Hibernate to see why aren’t my queries working, and the time I shot puzzled looks at Spring Dependency injection stuff to try and use it, but once I got things sorted, I never got back to them (until the next problem) – Every bit of tool know-how I can expect from someone is as far as a week’s worth of google, assuming someone around here hadn’t solved it already. By insisting on a specific tool I’m shooting my leg twice – I’m scaring away good candidates that lack knowledge of this tool, and I waste time on measuring whether people already know stuff I know I can teach easily, or expect them to investigate it on their own.
So, yes. I would rather use “reverse a string” as a discussion opener. It’s quick, it removes a lot of noise from the system fast, and keeps everyone focused and gives me insight to some of the basic programming skills (In about 30 minutes I usually get a decent assessment of: basic memory allocation understanding, the ability to trace code, test-case choosing, basic concurrency, naming conventions and programming habits).
If I want to see how well a person understands OOP, I would ask them to draw a small class diagram, or any other simple, targeted question.Data Structures? Possibly a search function, or something similar. Trying to figure that from a large piece of code, without seeing the person writing them in real time, is much more difficult.
I definitely appreciate the engagement here, Amit. You and I just have very different experiences and thoughts from them. Which is great, actually. I really like hearing how you articulate it.
I doubt we’re going to convince each other but you are certainly forcing me to think through my position more. Let me just take a few points of yours. I don’t want to risk cherry-picking and caricaturing your position so please know that I’m just picking out bits of contrast.
I must say that I don’t really get you when you say that “understand code” is vague
I was probably unclear. I just meant that if we say we want to see if a candidate “understands code” — that’s way too vague of a qualifier. I can understand a bit of code I’m looking at but still not know much about good coding. For example, a rejection of “fat models, thin controllers” is actually what I might want to see from a developer. Which is an abstraction above and beyond just “understanding code.”
As far as suggesting that using Selenium is just using the API – yes, that just about sums it up. All those cool features and ideas you mentioned? I DON’T CARE.
You may not. But many do. And for just about every project I’ve worked on, including the dozens of companies I’ve worked on them with, they cared as well. Those “cool features” are often how you use the API effectively and efficiently. For example, all the garbage that happened with Marionette / Geckodriver? None of the micro-frameworks I wrote for companies had even a hiccup with that. (Kinda proud of that one.)
Much of the stuff you mention are easily taught, and reinforced during code-reviews.
Sure. But as a candidate, I don’t have them in code reviews. I have them for a limited time where I have to see how they think, the kind of decisions they make, and why they make some rather than others.
And stuff like the effective use of a data builder or a context factory being taught is not so much the issue for me. Most anything can be taught. What I want to see, to some extent, is not just coding but conceptualizations. For example, if a test automater starts talking to me about “I keep the test data here in these files, because I don’t want it as part of my tests, but I needed a way to get the data into the code and I wanted that to be via friendly names” — well, they’re basically describing the data builder. They may not know the term, but they have an instinct for what the techniques that the term encapsulates.
If someone is choosing bad locators, or is not familiar with the best way to interact with our product when using web, I’ll be there to help and to comment about it during the code-review, or sooner.
Sure, but bad locators is extremely low-hanging fruit. You’ll notice I didn’t bring that up once in any example that I showed you. A better example is what happens when the locator changes dynamically. And that happens without a page refresh. And the element itself is unchanged. (Mutation observer anyone?)
I only care for optimizing what interrupts me. When I use sub-optimal scrolling strategies what do I lose? Execution time? Brittleness?
Example: I worked in an environment where the ads that were displayed had to be at least 25% on the screen for a certain fraction of time in order to count as a view of that ad. Just “scroll to” would not work. Neither would “position to element” in all cases, because it depended on where the position was situated. It might not be enough to get the element exactly 25% visible.
So what happened here when you used “sub-optimal scrolling” was automation that did not verify business value in a way that we could trust.
I respect the time of the candidates I’m interviewing. Asking for 4-5 hours of their free time before the interview is quite a lot.
Sure, it can be. But it also gives us and them time to really engage on something and put their best foot forward in a context that equalizes out some of the pressure.
I don’t want to disqualify people who are used to a different tech stack. The code you linked to tells you quite a lot, but for me it is worthless – I don’t speak Ruby, …
That’s why you let candidates pick the language. That could have been done in Python, Java, C#, JavaScript, etc. I had one person do it in Clojure. That was really interesting to me. I learned a lot, which was a great give-and-take on the interview. I love when candidates pick a language I don’t know very well or at all.
I’m not asking them to build everything from scratch …
Me either. That example I showed links for uses Watir which wraps Selenium. Nothing built from scratch except the logic specific for testing Amazon. Someone else might use WebdriverIO (JavaScript), Webium (Python), FluentLenium (Java), whatever. Some wrap it with RSpec, Mocha, Jasmine, Cucumber, etc.
If I want to see how well a person understands OOP, I would ask them to draw a small class diagram, or any other simple, targeted question.
Sure. I might frame this in a test context as draw me your class layout for a test framework. What’s it composed of? What are the interfaces? What’s hidden? What’s exposed? Where are the abstractions and how are they managed? (Particularly important if the candidate seems to like BDD or ATDD style solutions.)
Trying to figure that from a large piece of code, without seeing the person writing them in real time, is much more difficult.
I agree, it can be more difficult. I’ve found it ultimately more rewarding. Again, from a test solution developer standpoint. I can’t speak to it from an enterprise developer standpoint. I realize it’s possible to argue there should be no such distinction, at least in terms of the interview. I’m still convinced that there is such a distinction, hence my desire to reframe those types of interviews.
That said, I’m still willing to be convinced. And I certainly do agree that code interviews, of the sort you prefer, have their place. But as I said in a comment to another person, I’ve “passed” code interviews where I literally just memorized a bit of code and the explanation for it. I barely understood it. A reverse Polish notation parser comes to mind, for example.
Obviously that could be a fault of the interview process, I realize, since they could have dug in. But it’s much, much harder to fake an example like mine, particularly when you have to explain the decisions behind it.
One other thing I should probably add is that, as a test solution developer, you are a tester first, developer second. And I bring that up because one thing I like those exercises to do is force the candidate to make some decisions.
For example, this part here shows what I mean: notes on modifications.
What the candidate was told to automate was, in fact, not a good way to automate it. One of the products, in fact, couldn’t be used as it was (ebook only) and another was too vague. I’ve seen candidates provide extremely intricate code to handle both situations. What I really want to see is them question the requirements (that’s what testers do) and have some opinions about what should and should not be done (test smells) and then implement accordingly. There is even a concern in there about test data (live data we don’t control can go out of stock, making brittle tests).
All of these things are critical skills in a tester, particularly one that will translate those skills into automation — automation that, by definition, won’t think like the tester will. I don’t just need them to code; I need them to think about code that is executing test design.
Having them reverse a string or provide a solution that checks for the sum of a target in an array of numbers will tell me absolutely none of that.
(still funny commenting issues, it seems I can use the “text” edit mode, lets see what it will do)
First of all, thank you. You are very clear on value of the way you present, an it does make me think and evaluate those gains in my context.
If I try to be concise about it, it seems that there are several differences between us (please correct me if I understood you incorrectly).
1. We disagree about the level of insight that could be gained by an offline task. Your experience is that longer code that covers a lot of subjects provides you with a lot of insight, and is harder to game. My experience is almost the opposite – I believe that I gain more insight when a candidate is pseudo-coding on the whiteboard, as I get to see the way they think, the way they deal with surprises and I’m able to shift the focus of the exercise to match the questions I have about the candidate. For me, gaming the interview is not an issue – both because I’ve yet to see it happen, and because if someone is able to fake the level of details we get to, they can “fake” working with us just as well (I don’t think I could prepare someone to pass an interview with me). Also, when working offline, I know I’m reading the code someone produced with help of friends – I helped someone write a Selenium task a few years back. He did tell the interviewer that I helped, but he could just as well say nothing. Would such preparation save someone who’s coding illiterate? No. But it would cause all of us to lose time – me by reading the code and clearing the time for an interview, them by doing the exercise and coming to the interview.
2. Time – You seem to be more ok with asking candidates to invest time in your hiring process. I try to minimize that time where I can (and would be quite unhappy if I got 5 hours worth of homework, on top of the day off I’m taking to get to the interview). The point you raise about reducing performance pressure is quite interesting and I’ll have to think about it a bit more, as this is very valuable (not sure if it trumps out minimizing the fuss for the candidate, but it might).
3. You seem to have a very well defined problem space. For me, I don’t know what sort of problems I’ll face next month. Stuff like “how to interact with the product” or “how to manage our test data” are solved problems for us (not in a perfect way, but in a good-enough-for-now way). Therefore, the deep knowledge
4. It seems that we differ a lot on what we are willing to put for “on the job training”. My approach for that is “just give me someone that can code, and we’ll go together through with the rest. We had people with experience in testing hardware (LCD screens, I think), with embedded software, with AI, and without any prior experience – having a home-test around selenium would disqualify them without reason.
The last part leads me to wonder – how easy is it for you to find candidates? In my environment it’s quite difficult to find good testers that also code, and putting extra barriers is not something we can afford.
What are the things you are willing to leave for on-the job training?
Do you have a way for a candidate that says “I’m a great tester and a decent coder, but have never worked with Selenium before”, or is it something you would consider as a reason not to hire?
Also, I wonder – Can putting such emphasis on Selenium when interviewing send out the wrong message? Ideally, we would want our automation to cover up a lot more – database, APIs, maybe benchmarking. Is the focus on this one tool sending a message that “here we do a lot of Selenium”? The message I want to send is “we just get the job done the best we can, and we don’t care how exactly”.
(still funny commenting issues, it seems I can use the “text” edit mode, lets see what it will do)
Sorry, yeah, I set it to just one thread inwards because otherwise it compresses the text ridiculously. I need to work in a better commenting system.
1. We disagree about the level of insight that could be gained by an offline task. Your experience is that longer code that covers a lot of subjects provides you with a lot of insight, and is harder to game.
I don’t mind coding in person and I have had people write code with me as we were sitting there. But longer code shows me more decisions they make about how things hang together. And test solutions are all about that. It doesn’t have to be Selenium. It can be Appium. It can be with a REST client. It can be with Pact. Or with numerous other aspects that are focused on testing. Generally the main things I have automated testers working on, when not at the code level, are browser-based tests, application (mobile)-based tests, and integrated (not integration!) tests. In fact, knowing that there is a distinction between those latter is pretty key for a test developer.
For testers, regarding the way they think, I much more need to see how they explore and investigate, including the assumptions they made. Code rarely helps me with that, except in a very limited context. One of the best ways to do that is with a game challenge. I wrote “Test Quest” for just that purpose.
2. Time – You seem to be more ok with asking candidates to invest time in your hiring process.
Yes, just as we are investing time in them. But I do agree: this can become too much. But part of the code challenges are based on the likely skill level, given their time in the industry or what they present as part of their introduction to the company. For example, if an automation candidate shows me a resume with their GitHub and I see they’ve written a lot of automation or practiced with various frameworks or solutions (not just Selenium), I may tailor a challenge to that.
Sometimes I have them come in and say: “Show me what your most proud of on your GitHub. Take me through it. How does somebody use it.” Those kinds of situations have been where I found some of the best people.
3. You seem to have a very well defined problem space. For me, I don’t know what sort of problems I’ll face next month. Stuff like “how to interact with the product” or “how to manage our test data” are solved problems for us (not in a perfect way, but in a good-enough-for-now way).
Right. But … solved for you, isn’t it? It might be solved for me too but … I don’t know if it’s “solved” for the candidate. I don’t even know if they consider test data to be a “problem” worth solving. I’ve met candidates that don’t know what “gold data” is, for example. Or don’t understand injecting data via API or DB calls. Who don’t know what a data builder pattern is. I’ve seen testers who don’t know what “incidentals” mean when it comes to test data. So while we in the company feel we have solved that, I want to see what the candidate has been exposed to and/or thought about.
4. It seems that we differ a lot on what we are willing to put for “on the job training”. My approach for that is “just give me someone that can code, and we’ll go together through with the rest.
With testers, I can’t rely on just if they can code. I can’t train as easily things like good use of the scientific method; good use of exploration; good investigative skills; good use of encoding exploration as scripts. Another big one for test solutions is the appropriate use of abstractions. I don’t worry if the person knows every aspect of Appium or Selenium or Pact or whatever other tool is out there. That I can train for.
Now if I’m just hiring an “automater” — meaning someone who literally is just going to be looking at code and writing code — then I get into that question of do I want “a developer who can test” or “a tester who can develop.” It’s pretty interesting that people can’t often make that distinction but it’s there and it’s very important. I don’t hire “just an automater” because that would be turning testing into a programming problem, which is not. So what I can train for is a bit nuanced.
The last part leads me to wonder – how easy is it for you to find candidates?
That can code? If their focus has been automation, this is generally pretty easy. And here I mean “can code” in a way that is relevant to what they will be doing 90%, if not more, of the time. But finding people that can code and that understand test code is itself code that has no tests and act accordingly: trickier. And having automated testers that don’t reduce testing solely to a programming problem? Harder. Finding a good generalist with specialist tendencies. Even harder. Even with automation, we’re often in a polyglot environment. We’re using security tools, database tools, web browser tools, mobile app tools, consumer-driven contracts, property tests — and some of those may be in different languages. An automater needs to be able to move between those with a certain amount of fluidity. We might be leveraging business and thus using BDD solutions which add complexities, even to the coding, that many testers aren’t aware of — until they end up with an unmanageable mess.
It doesn’t answer question, perhaps, but I talk about looking for and valuing the modern tester. I want to see technical testers that have engaged with the which language dilemma.
So maybe it comes down to this: if someone truly just wants “a developer who can test”, then hire for a developer. Because what that often means is just someone who is writing automation without the wider ambit of testing being front and center. The “who can test” part will likely not be as important to that company as the “developer” part. So hire a developer and do whatever you want with those code challenges. But I’m talking about technical testers — and that’s a different thing. That’s “a tester who has some development background.” That background is secondary to them being a tester first and foremost. That’s the kind of interview that I think needs to be done different from a code challenge perspective.
Well, things are getting very interesting…
I’ll pick up on some of the things you say
With testers, I can’t rely on just if they can code. I can’t train as easily things like good use of the scientific method; good use of exploration; good investigative skills;
You are, of course, 100% right on that. I was assuming a capable tester when I wrote this sentence, I should have said that clearly.
Right. But … solved for you, isn’t it? It might be solved for me too but … I don’t know if it’s “solved” for the candidate.
Well, if it’s solved – why does it need to be solved for the candidate? Does it need re-solving? I can show new employees around and tell them “this is how we manage data” or “this is how we make sure we don’t need to touch the Selenium code in 99% of the times”. Being someone who got to an existing place with no prior experience but with a decent framework I can attest that those problems become evident the moment you try and do something outside the framework, and once it’s visible, noticing the pros and cons of the choices that were done in the framework one works with is quite easy to figure out.
I don’t hire “just an automater”
Yep, that’s why I keep using “tester”. Personally, I never understood places that have “test automation” as a separate role than “test”. When I say tester, I assume both testing skills and coding skills (so, someone that lacks either will be defined specifically, such as “A tester with no mobile experience”)
Even with automation, we’re often in a polyglot environment. We’re using security tools, database tools, web browser tools, mobile app tools, consumer-driven contracts, property tests — and some of those may be in different languages. An automater needs to be able to move between those with a certain amount of fluidity.
Now I’m even more confused – with such a vast array of tools you use on a regular basis, what does a test around one tool give you? What does it give that is worth the chance a candidate will judge you and say “oh, they are one of those places that think ‘automation’ is driving the GUI, I think I’ll pass”?
I don’t worry if the person knows every aspect of Appium or Selenium or Pact or whatever other tool is out there. That I can train for.
I did an interview training recently, and one of the key takeaways for me was to separate the skills we’re looking for in a candidate to quadrants set by “Is it a skill we must have?” and “can we teach it on the job?” and then ask only for the skills in the must-have+can’t-teach quadrants. If you can train for this, why waste time with it?
Well, if it’s solved – why does it need to be solved for the candidate? Does it need re-solving?
“Solved” in a loose sense. Like I said, I more want to see if they’ve even engaged with the problem. Maybe they’ve come up with interesting ways that we haven’t thought of based on past experience. But even if not, I like to see what someone recognizes as a problem, how they articulate it as a problem, and what they have done to provide solutions. For example, someone asked me about “expressing data conditions in tests” and they were surprised when I showed them my usage of a data builder. The second code snippet there, with the line
on_view(Warp).using(data_about('alpha centauri')). Really simple to implement but, for them, that was a very different way of expressiveness in tests, which lead to discussions about where I stored “alpha centauri”, how that went from test data to test condition, how people (like business) could just input another data condition (like “epsilon eridani”) and so on.Now I’m even more confused – with such a vast array of tools you use on a regular basis, what does a test around one tool give you?
It’s not a (candidate) test around a tool, per se. It’s using one particular tool to test with and seeing how someone uses their tester mind to engage with a technology solution. The test thinking that goes into Selenium is very similar to that which goes into Appium to that which goes into Pact. Not the coding; but the test thinking. At a high-level, mind. One of the commonalities is the expression and abstraction of test conditions and data conditions. (Note the latter is not the same as “test data”, of course.)
By the same logic, arguably at least, you could ask: what does one programming problem tell you for a candidate? I might have given them one they memorized already. (I’ve passed coding interviews with challenges that I barely understood. I had just memorized the solution and the explanation for it or was able to look it up.) Or I might have given them one that is particularly easy for them while another one might have ground them to dust.
What does it give that is worth the chance a candidate will judge you and say “oh, they are one of those places that think ‘automation’ is driving the GUI, I think I’ll pass”?
If the candidate thought that, that would mean they probably weren’t listening when we told them we test at multiple levels of abstraction, using multiple tools that suit the needs for testing at that level. We encode those tests in various formats for those different levels. The candidate challenge (perhaps Selenium, perhaps API) is for one of those formats and — in the case of UI-focused — it’s a challenge that’s probably one of the more common in terms of familiarity but also fraught with the most chances to make bad decisions. It’s the situation where testers make a lot of assumptions about usage (and requirements). After all, this is the UI — it’s what most of our user-base sees when they interact with our product. It’s the situation where testers are the most likely to encounter problems that lead them to write brittle tests or flaky tests.
That’s not to say you can’t make horrible decisions in API tests or in property tests or in database tests. But we generally don’t have time to test for all. So we look for the one that has demonstrably proven to be a fertile ground for making not just a series of errors, but a series of errors that also bring to light assumptions and biases the person might have regarding what is a good and a bad test, including the level of expression of that test.
If you can train for this, why waste time with it?
Sure, but I can train for coding. All coding. Given enough time. So why waste time with any code challenges on the interview? I’m not being obtuse there. It’s the thinking that you’re actually looking for, as you indicated earlier. The thinking is taking place in the context of engaging with some code to solve and/or implement a particular problem/algorithm/whatever. The same applies for what I’ve been talking about.
I think we’re getting somewhere, so I’ll try to push it one step towards the “meta” level. I’ll start with the last thing – training people. You mentioned rightfully that you can train candidates to code, given enough time, are you willing to spend that amount of time? If you do – you don’t need to check coding skills. If you don’t, you can’t really teach that (because you don’t have the time to do it). Being able to train people means that you both know how to do this, and is willing to spend the resources required to do so. So, in this case, “don’t want to” equals “can’t”.
Which brings me back to “solved problems” – you are completely correct about the value of a different insight, but let’s say someone is not familiar with one (or all) of those problems – what is it you will need to train them in, and to what end? What is the core competency you can’t train? If someone doesn’t understand OOP properly, for instance, I know this person wouldn’t be able to contribute decent code, will have harder time to understand how our code is structured and code-reviews should be three times longer in order to be effective until the concept has sunk in, which is time I am not willing to spend. When a new employee got to do a Selenium task (with no prior experience in it), I took 10 minutes to describe the idea of language-binding->wire protocol->webdriver->browser, pointed and said “here are some examples of how we use it, and don’t you dare putting selenium code directly in the tests” and sent them to do the task. It took slightly longer than it would take me to do – but the result was satisfactory, and the resources invested were acceptable. Given that you are willing to teach any of the tools you use (otherwise, you would have put them all in your interview process), what is it that you are not willing to train and is revealed only on such tests?
What does one, simple, programming question tell me?
Well, not as much as one can get from a larger piece of code, but that’s the nice thing about it – I’m not distracted by excessive data. I see answers to very targeted questions. If a candidate has horrible naming conventions, that would probably distract me from the fact that they use interesting ways to manage their test selection, or that they are very proficient with a tool. Asking short, focused questions (notice the plural form), I can focus on what is important – one question to check coding style and basic algorithms, another to check for code design, another to check data structures understanding, and yet another for whatever I need.
Finally,
The thinking is taking place in the context of engaging with some code to solve and/or implement a particular problem/algorithm/whatever. The same applies for what I’ve been talking about.
That’s where I’m aiming – In my experience, teaching a decent coder the thought process required for writing “automated tests” is a rather easy process: Data management is a problem that exists in all programming, testing is simply putting a different set of constraints around it, and the same is true for most other challenges that are unique for coding tests: They are very similar to other programming problems, and once you state them explicitly, it doesn’t take long for a good coder to adapt. What is the state of mind that your experience shows is too expensive to train for? How would someone missing that would e ineffective in your work?
(Also, one thing that I’m assuming, and probably is quite important to say out explicitly: I’m assuming a tester who will work in a team and will not have to invent the wheel. When hiring for a consulting position, or any other position that will involve working alone a lot – I completely agree with your approach of recruiting)
Which brings me back to “solved problems” – you are completely correct about the value of a different insight, but let’s say someone is not familiar with one (or all) of those problems – what is it you will need to train them in, and to what end? What is the core competency you can’t train?
Okay, so far as the test data, let’s say they aren’t familiar with that at all. Depending on their skill level and where they are in their career, that may concern me more or less. That’s a key thing here. All of this has to be taken into account with the notion that, at a certain point in our careers in a certain discipline, we do expect people to know things that those more junior do not. I would imagine a developer might better expect design pattern understanding from a senior type developer rather than a junior, assuming everyone agrees on what those terms mean.
But would that lack of, say, test data thinking concern me such that I wouldn’t consider someone if they didn’t have this? Not necessarily and, in fact, has not in many cases. If I saw them not consider test data as part of a coding example, let’s say, I would simply ask them about it. For example, “I notice you put in very specific values here. What do you think about that choice? Are there dangers to that?” Etc. I more want to see how they engage with that idea. How they would code up a solution to implement that idea is of much less concern to me.
Let’s take another example that provides some interesting boundaries, such using BDD solutions, which can (but does not have to) sit in front of many tools, not just Selenium. This is using a test description language. I have devoted a whole category of posts to those. Can I train someone on all of that? I definitely can. But it takes a (perhaps surprising?) amount of time because it becomes a very precise way of specifying test and data conditions, wrapped around statements of requirements, often bracketed by feature injection. I can train for that. Ultimately. But it takes a lot of time to reorient people if they have never been exposed to that or if, quite frankly, their ability to write concisely and operationally is lacking.
Now let’s take it a step further. Those TDL-styles solutions (BDD type approaches) will often delegate down to code. That might be Selenium but it might also be to database logic or to APIs or to contract-based solutions, like Pact. There are ton of ways to go really wrong about this delegation part. As just one example, and admittedly the more complex, you have the English (feature files), the step definitions (that usually tokenize the English based on regex), and the models (page objects, screen objects, etc) that you delegate down to and those models, in turn, probably call and/or utilize Selenium, Appium, REST client, etc. Lots of layers there. Lots of abstractions.
I can train someone on all of that too, including how to avoid being too heavy at any one level of abstraction. Obviously the more they’ve been exposed to it, the less I have to train. But I’m okay with training them on that — as long as I have some evidence that they can move between abstractions.
So, of those two (BDD and tech-behind-BDD), if we were implementing BDD-based automation and we already had a solution in place (a “solved problem” for now), what would I need to see from the candidate? I would actually more want to see examples of the TDL writing from them. I give a real (but anonymized) example of why this kind of thing is very hard in spec workhops and test writing, specifically starting at the part under the heading “THE DIALOGUE.”
As another example, there is a lot potentially wrong with this set of tests. Those delegate down to code (steps and model). But even without the code part at all, some testers can struggle to see what’s wrong with those tests as they are. So I just need to see that they can think about and reason about these different levels of abstraction and start to form opinions about what is going wrong, what could go wrong, what seems to make sense, etc.
Honestly, what’s told me the most about testers is what I talk about in reframing test interviews with gamification. Those are some core competencies of experienced testers that I need to see because that’s where I’ve seen teams fall down the most. Not at all on the coding aspects.
In fact, if I had my choice and going with the idea that I can always train on something like programming, I would forgo any and all code challenges — even for a technical tester — and have exercises that focus on what I talk about there.
That’s sort of where I’m at with this. When you are a hiring a “tester who has to do some development”, I can easily train on the development part, as you indicated and which I fully agree with. I’ve run intern programs where people with little to no coding experience were writing automation very quickly in a language they hadn’t ever used before or even seen before. A large part of that was because, as you indicated, we “solved” that part: we provide an expressive framework. That’s the test execution part. There’s a part where the “execution part” must consume the test design part — and that’s the part that I more want to see.
So when I say the technical test interview is broken, notice I did call it the “technical test” interview. I didn’t say the developer interview or even the code-based challenge interview in some wider sense, at least for my purposes. The problem is being a “technical tester” is often equated with simply being a tester that can code. And even then the “can code” part is not treated as contextually relevant for the type of coding they will be doing.
Ok, one second here.
You make some really good arguments, but something has just downed on me – Is this coding test your way to (also) check the way people think about testing problems such as questioning the requirements or test design and selection?
If so, it’s a small wonder I found this approach to be an absolute overkill.
The longer we go, it seems to me we seek pretty much the same (though I would guess my experience is leaning more towards interviewing junior positions), and the main difference is that you construct a single gigantic question that is intended to provide insight about everything you wish to know, where I prefer smaller questions, dealing with specific skills that are on my “must” list.
Is my guess correct?
Is this coding test your way to (also) check the way people think about testing problems such as questioning the requirements or test design and selection?
In short, yes. The automation I’m mostly dealing with is the type whereby it is written by a tester. So not unit-based, for example. And probably not integration-based. Integrated and system — definitely. Acceptance — most definitely. Because it’s written by a tester, and is in service to testing as a discipline, that means it must intersect with previous test design. And test design must intersect with some specification (implicit or explicit) that guided test design based on the value something was supposed to deliver. That value may be delivered in multiple ways, such as by a consumer-facing API as well as a graphical user interface (which may have to display on web and mobile).
So if I have to provide a coding challenge to testers, that’s what I meant by they have to be a tester first, coder second. I have no doubt they can learn to write code or I can train them to write code. But what about code that sits at multiple levels of intersection? Code that has to support potentially multiple audiences, including non-technical people.
The longer we go, it seems to me we seek pretty much the same (though I would guess my experience is leaning more towards interviewing junior positions), and the main difference is that you construct a single gigantic question that is intended to provide insight about everything you wish to know, where I prefer smaller questions, dealing with specific skills that are on my “must” list. Is my guess correct?
I think you are probably correct. I generally need to see someone making the broadest possible set of mistakes which means I need to give them the opportunity to do so. I need to see what traps they fall into (or avoid) as test design becomes test execution and (sometimes, but not always) becomes test code. If you are hiring a test solution developer — a “technical tester”, as opposed to just “a developer who writes automation” or “a tester who writes automation” — then it’s critical to be able to ferret out as much of this as you can in the limited amount of time that you have. And, of course, all of this has to be adjusted for relative skill level, such as expectations for a “junior” versus “senior.” Given my druthers, I would rather hire a tester and train them up on coding, rather than hire a coder and have to train them up on all of testing.
Thank you for the very interesting discussion – while I still wouldn’t choose your way as an interview strategy, I think I now understand better the reasoning behind it, and see how it might work for some people.
I really enjoyed the discussion here, it really forced me to think about both the way I conduct interviews and to compare it with your very well presented approach.
To you as well! I appreciate you engaging as you did. You forced me to better articulate my thoughts and you certainly gave me the chance to question my own thoughts in light of your own thoughtful objections and desire for clarification. While I still feel a certain confidence in my approach on this, you have given me a healthy counterpoint that I need to think about more and see if I’m being too rigid and inflexible in my thinking.
I was entirely set against your point of view until I saw your responses to Amit. Then it all clicked for me. One of the challenges I have is that so many of the coding challenges are known about these days, that it takes very little effort to just look something up on GitHub or Stack Overflow or one of the numerous “Coding Interview” sites.
The only way to alleviate that concern of someone looking it up is to have them do it all in person. But I’ve had people want to look things up in the interview, asking me if they could. And I’ve been torn on that. If I let them do that, why didn’t I just have them do this at home as part of CoderPad or something? But if I don’t let them do that, it’s not reflective of the environment we are, which is where “If you don’t know, admit that, and look it up.”
If someone said to me: “You can’t look this up” my response would likely be of the form: “Ah. So I can’t do what EVERY single efficient programmer does ALL the time that shows an ability to effectively find information without wasting time?” This would likely negate my candidacy but, then again, I wouldn’t care because I don’t need someone micromanaging how I solve problems.
For me, when I’ve had to do in-person coding exercises with a candidate, I’ve had a laptop ready that we could look things up on and, in fact, encouraged that. If you don’t know — you don’t know. Fine with me. But how do you go about finding out? Once the candidate found something that seemed promising, we’d try it. Clearly since they didn’t know how to do it to begin with, they probably don’t immediately know why the solution works. So I spend time talking with them about that as well. We’ll do a bit of code archaeology.
As programmers, you often spend as much, if not more, time reading code than writing it anyway, so the ability to do this could (arguably) be just as important.
More to the point, this approach allows me to see the person’s personality, see how they engage with a problem, and see how they work to understand that problem once they are provided with an “answer.” Because, as we know, just having an answer doesn’t mean you have understanding.
Just read through the article and the comments. As a newbie automated tester with very rusty coding skills I found this food for thought. It was also very refreshing to follow a proper discussion where both sides were able to voice their opinions instead of bashing each other online. Thank you for that.
Good article, one thing i don’t like is calling them Automated tester/ software developer in Test. I don’t understand that terminology. For me if you are hiring some one to automate your checks better hire a developer. if you are hiring a tester look for a tester who has testing skills e.g. people who can think of exploratory testing. A tester with coding knowledge will help to automate some of his tasks to get to the point where he can start testing e.g if he has skills to automate the test data setup so that he don’t need to spend time on that every time.
Yeah, the notion of “Software Development Engineer in Test” (SDET) or “Software Engineer in Test” (SET) has become an interesting set of elements that our industry is currently pivoting about. The very notion of a distinction between the SDET and the SET itself is interesting, the difference of course being the lack of the word “Development” in one of the titles. It’s been interesting to hear companies talk about how they make that distinction or what it even means to them.
That said, regardless of what they are called, if someone is being hired to provide automation, I don’t think that has to be a developer, per se. What it has to be is someone that has development skills. That can be — and I would argue, should be — a tester. Certainly some coding skills are important there but, like developers, it’s more than just being able to code. It’s being able to code in a specific context; in this case, the creation and/or use of frameworks or micro-frameworks for automation. The bare minimum is simply being able to use such frameworks. If you start getting into the creation of test supporting tools, I suppose “Test Solution Developer” isn’t a horrible title and I have certainly been known to take that title for myself, even when I had an “official” one.
That context I describe, however, is, in my opinion, what more needs to be sought in so-called “technical tester” interviews. My ability to reverse a string, or sum an array, or display a number according to some base may tell someone something. I don’t doubt that. But it won’t tell them what they most need to be learning about me. Testing, as a discipline, is often about helping people reframe questions so they get answers they need, rather than the ones they think they want. I see this whole technical interview process as an example of that, just on a more massive scale.