Automation Language Is Not (Necessarily) Your Development Language

Written by Jeff Nyman 27 December 2015 2 Comments

Way too many companies out there have this notion that testers must use the language that their applications are written in. So if your company writes an application in Java, that’s what the testing solutions should be written in. This is laughably wrong and completely short-sighted. It’s what you tend to get when development managers are responsible for building up test teams or when you have test teams that have to rely on others to do their work (and their thinking) for them.

Wow. Now that initial comment was a bit of a broad generalization, no? It was and it’s probably a bit unfair. But I do see this kind of thinking a lot. And it never ceases to be annoying. Now — having said this — let me be honest here and say that I have flirted with these thoughts myself. I asked if you should align your test programming language with the development language. And if you read that post, it might seem like I was coming down on the side of “yes” but, in fact, if you read to the bitter end, you’ll see I was experimenting with my thinking and my practice.

A lot of this experimenting was based on one of the most common questions I get asked from fellow testers, which is the “dilemma” of which language should I learn?. Others get asked this as well, as you can imagine. Joe has a good post about this on which language to learn.

The Language Choice

Keep in mind here we’re talking about the programming language you use in your automation, not the particular tool that you choose. Choosing a tool is a whole topic in and of itself but it’s not entirely orthogonal to the programming language question. Here, however, I’m focusing on the idea of the programming language for testing needing to be the same as that used for programming the application that is being tested.

Testers and Developers: Working Together

Alister Scott had some thoughts on this topic regarding choosing a language for automated acceptance tests. I’ll quote one bit:

Even if the software testers are responsible for writing and maintaining the automated acceptance tests, having them in the same language the programmers use will mean that the programmers can provide support for any issues the testers have, and are more likely to collaborate with the testers on these. The testers also pick up knowledge of the language used for the core application which means they are more likely to be able to fix bugs that they find.

That part I largely disagree with. I often hear that sentiment, however. “But if you just go with this language, then the developers can help write your tests!” Sounds great in theory. (Or does it? That’s debatable.) But I rarely see this happen. And I rarely want this to happen. Developers have plenty of their own work to do and the last thing they need is to be given more. But Alister is right to call out a specific type of test — what he calls “automated acceptance tests” — because the level you are testing at matters.

Automation is done at various levels but the two main focal elements people think of is at the code level (so-called unit testing) and at the UI level (system testing, for lack of a better term, and sometimes “acceptance testing”). It’s more nuanced than that but my problem is that when you get developers helping along with higher-level tests, they often write all those tests as if they were writing unit tests.

Unless developers understand that automated checks need to be human readable to improve communication, they will tend to automate them in a way that minimizes development effort. When developers focus on higher-levels of test automation, you’ll generally find that you don’t get better collaboration even when you get more communication. The reason for this is because tests should serve various audiences. Even the automated checks that are done to reinforce testing. With developers working on this, you’ll tend to find that the approach often leads to tests that are too technical and that are scripts rather than specifications.

Testers and developers can work together just fine but I think we each have plenty of work in our respective domains that we don’t need to be doing each others.

Testers Fixing Bugs?

Alister also says that a positive may be that testers pick up knowledge of the language of the core application. Probably a good thing in general. But then, in Alister’s view (and that of others who share his view), those testers are now more likely to be able to fix the bugs that they find. But is that a good thing? Do developers want me doing this? Don’t I have enough to focus on with quality assurance and testing without adding fixing bugs to the list?

I can have input into the bugs but here is where, in my view, we get into one of the more pernicious problems in our industry: the conflation of testing and development activities. It’s a tricky balance and I talk a bit about this in my “back to testing’s roots” post as well as discussing how we value the modern tester.

But, to be sure, if I most want to fix the bugs, I’ll do my best to prevent them from getting in there in the first place, which means treating testing as a design activity. Which is a whole other aspect to my discipline I need to focus on — which I won’t be doing if I’m fixing the bugs I find.

Testers Reimplementing Functionality

Adam brings up some interesting points as well in his “but our application is written in ‘x’ myth” post. Quoting one bit:

There is actually one really good reason to write your scripts in the same language as the application itself. And that is when there is a ridiculously complex algorithm implemented in that language that you want to leverage as part of your verification steps.

Obviously this depends on what “ridiculously complex” means and that will depend on context but I can say that I have worked in plenty of environments (ad serving, hedge fund trading, clinical trials, banking and health care) and I have either (a) never had to re-implement complex algorithms in my tests or (b) I could use the many ways that languages can talk to each other. For example, in one context I used JRuby with my Ruby-based test solution to talk to a Java algorithm for serving ads.

Just to drive this point home, I want to show an example of this. I have a fake application called Symbiote that I use to test out my own automated solutions. One of those pages has a Stardate Calculator on it. That Stardate Calculator is triggered by JavaScript and the conversion of a particular stardate into a calendar date is also done via JavaScript. When I test this, however, I don’t have to use JavaScript. I can reimplement the date conversion in another language just as well as I could in JavaScript. Here’s an example of some Java logic that reimplements the date conversion portion as part of a test:

public void convertTNGStardate(double value) {
    enableForm.click();
    tngEra.click();

    stardate.clear();
    stardate.sendKeys(Double.toString(value));

    convert.click();

    String result = calendarDate.getAttribute("value");

    Calendar calendar = modelTNGCalculation(value);

    int year = calendar.get(Calendar.YEAR);
    String month = new SimpleDateFormat("MMM").format(calendar.getTime());
    int date = calendar.get(Calendar.DATE);

    assertThat("Year was not valid", result, containsString(Integer.toString(year)));
    assertThat("Month was not valid", result, containsString(month));
    assertThat("Date was not valid", result, containsString(Integer.toString(date)));
}

private static Calendar modelTNGCalculation(double value) {
    double stardatesPerYear = value * 34367056.4;
    DateTimeFormatter dtf = DateTimeFormatter.ofPattern("MMMM d, yyyy HH:mm:ss");
    Date origin = Date.from(LocalDateTime.parse("July 5, 2318 12:00:00", dtf).toInstant(ZoneOffset.UTC));
    double milliseconds = origin.getTime() + stardatesPerYear;
    Date dateResult = new Date();
    dateResult.setTime((long)milliseconds);

    Calendar calendar = Calendar.getInstance();
    calendar.setTime(dateResult);
    return calendar;
}

public void convertTNGStardate(double value) {

enableForm.click();

tngEra.click();

stardate.clear();

stardate.sendKeys(Double.toString(value));

convert.click();

String result = calendarDate.getAttribute("value");

Calendar calendar = modelTNGCalculation(value);

int year = calendar.get(Calendar.YEAR);

String month = new SimpleDateFormat("MMM").format(calendar.getTime());

int date = calendar.get(Calendar.DATE);

assertThat("Year was not valid", result, containsString(Integer.toString(year)));

assertThat("Month was not valid", result, containsString(month));

assertThat("Date was not valid", result, containsString(Integer.toString(date)));

}

private static Calendar modelTNGCalculation(double value) {

double stardatesPerYear = value * 34367056.4;

DateTimeFormatter dtf = DateTimeFormatter.ofPattern("MMMM d, yyyy HH:mm:ss");

Date origin = Date.from(LocalDateTime.parse("July 5, 2318 12:00:00", dtf).toInstant(ZoneOffset.UTC));

double milliseconds = origin.getTime() + stardatesPerYear;

Date dateResult = new Date();

dateResult.setTime((long)milliseconds);

Calendar calendar = Calendar.getInstance();

calendar.setTime(dateResult);

return calendar;

}

The key line there is line 12 in convertTNGStardate which calls down to a model calculation. The method modelTNGCalculation does just that: it essentially reimplements a JavaScript algorithm in Java in order to provide a specific date to test against. I actually showcased another example of this with my Specify tool in the post models, rules, and features where I implemented a specific algorithm for calculating the surface gravity of a planet to help determine if a “Planet Weight Calculator” was working. In that case the planet weight calculations were similarly in JavaScript but I was modeling them in Ruby.

In fact, if I’m going this “re-implement an algorithm” route, there’s a good argument to be made for using a different language and that argument is the same you would hear in a math class: solve the same problem in a different way to make sure you get the same answer.

Pick the Language Based on Ownership

It really can be this simple: to determine which language to go with, answer one particular question. Who will own the automation? Not just own the execution of it but its maintenance as well as future development. Your answer to this should help you more effectively choose a programming language for automation.

One argument that comes up in this context is “Well, that’s all good and fine — but it’s easier to hire Java/Python/C#/whatever developers.” That can be very true. You may be in an area or market where hiring developers, including test solution developers, is much easier with a given language. If that’s the case, then as you build your team, you will logically hire people that are more apt to work in the language you want. And as long as that team agrees, you have no issue with that language choice.

But sometimes you have people who want to go a different route. This may be because they are more familiar with a particular language. And familiarity is nothing to scoff at. There’s nothing wrong with taking time to learn a new language but keep in mind there’s the tradeoff of what you’re paying people for. My guess is you would rather have working test solutions that you trust in a faster timeframe than people struggling to put together such solutions in a language they don’t know yet.

You may also have the case where testers are perfectly familiar with the language of the developers (say, Python) but would prefer to rely on a language that has a much wider and robust test ecosystem (say, Ruby) or they would prefer to rely on an ecosystem that can utilize many different languages (say, the JVM, leveraging Java, Groovy, and Scala).

However, let’s put all that aside for a second and talk about how bad any putative “language divide” truly is. Here’s an example of the automated script in two different languages:

Python

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("http://symbiote-app.herokuapp.com/")

open_form = driver.find_element_by_id("open")
username = driver.find_element_by_id("username")

open_form.click()
username.send_keys("admin")

from selenium import webdriver

driver = webdriver.Firefox()

driver.get("http://symbiote-app.herokuapp.com/")

open_form = driver.find_element_by_id("open")

username = driver.find_element_by_id("username")

open_form.click()

username.send_keys("admin")

Ruby

require "selenium-webdriver"

driver = Selenium::WebDriver.for :firefox
driver.navigate.to "http://symbiote-app.herokuapp.com/"

open_form = driver.find_element(:id, open)
username = driver.find_element(:id, 'username')

open_form.click
username.send_keys "admin"

require "selenium-webdriver"

driver = Selenium::WebDriver.for :firefox

driver.navigate.to "http://symbiote-app.herokuapp.com/"

open_form = driver.find_element(:id, open)

username = driver.find_element(:id, 'username')

open_form.click

username.send_keys "admin"

Someone could come in and as long as they learn the Selenium API — which is what I’m using here — could easily start utilizing either Python or Ruby to get up to speed. After all, you don’t have to master all of Ruby or all of Python, just enough to utilize the Selenium API. And there are copious resources out there that help with that. Now let’s consider non-dynamic languages with that same script.

using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI;

class Symbiote
{
  static void Main()
  {
    IWebDriver driver = new FirefoxDriver();
    driver.Navigate().GoToUrl("http://symbiote-app.herokuapp.com/");
	
    IWebElement open_form = driver.FindElement(By.Id("open"));
    IWebElement username = driver.FindElement(By.Id("username"));
	
    open_form.click();
    username.SendKeys("admin");
  }
}

using System;

using OpenQA.Selenium;

using OpenQA.Selenium.Firefox;

using OpenQA.Selenium.Support.UI;

class Symbiote

{

static void Main()

{

IWebDriver driver = new FirefoxDriver();

driver.Navigate().GoToUrl("http://symbiote-app.herokuapp.com/");

IWebElement open_form = driver.FindElement(By.Id("open"));

IWebElement username = driver.FindElement(By.Id("username"));

open_form.click();

username.SendKeys("admin");

}

Java

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;

public class Symbiote {
  public static void main(String[] args) {
    WebDriver driver = new FirefoxDriver();
    driver.get("http://symbiote-app.herokuapp.com/");
	
    WebElement open_form = driver.findElement(By.id("open"));
    WebElement username = driver.findElement(By.id("username"));
	
    open_form.click();
    username.sendKeys("admin");
  }
}

import org.openqa.selenium.By;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.WebElement;

import org.openqa.selenium.firefox.FirefoxDriver;

public class Symbiote {

public static void main(String[] args) {

WebDriver driver = new FirefoxDriver();

driver.get("http://symbiote-app.herokuapp.com/");

WebElement open_form = driver.findElement(By.id("open"));

WebElement username = driver.findElement(By.id("username"));

open_form.click();

username.sendKeys("admin");

}

Again, these two languages look very similar in their constructs. If you remove the boilerplate elements necessary for each language, you’ll find the core parts of the logic pretty much look just like the dynamic language counterparts.

Automation Relies on Existing APIs

So let’s consider something that’s often obvious in retrospect but often not when people are arguing about what language to use: if you are automating web applications or services, you will probably be working with Selenium to some extent, regardless of your language of choice. And Selenium has an API.

In fact, Selenium is an API. Selenium is an API that allows you to write code to drive a browser. So Selenium is a browser automation tool, but Selenium is not a test automation tool and not even a checking tool. Selenium doesn’t even provide a way to determine if your test/check worked. You specifically have to use assertion or expectation libraries to figure out if what you automated did something you expected.

The Selenium API talks to native implementations and binaries via a wire protocol. This wire protocol is a RESTful web service that uses JSON over HTTP. That’s the basis of Selenium. I’m belaboring these details a bit because I want to drill home the fact that none of this has anything to do with you automating your actions on your site or application.

So in order to work well with Selenium you need to choose a programming language to write your tests in. And as I’ve shown above, the means to do this are available in all mainstream languages. You can also do the same exact things in JavaScript, Go, PHP, Perl, Haskell, and others.

Automation Relies on Ecosystems

I mentioned above a consideration on the wider ecosystem that you want to operate in. It’s generally well-known that the Ruby community, for example, is very “test infected,” to use an overused term. This means that the community as a whole places high emphasis on various degrees of test thinking and test solution development. That has manifested in many test solutions being available. That means you not only have a lot to choose from but a lot to learn from. This means that you’ll often find many “wrappers” around existing tools and APIs, which can help streamline your automation development tasks.

Another consideration is the operating aspects of the ecosystem. I mentioned the Java Virtual Machine (JVM) which is a fantastic platform on which to build solutions because it’s so widely available and so well-supported. That said, test solutions in Java tend to be very baroque and annoying to use, and this is largely due to constraints put on you by the fact that you are using the JVM. However, that’s Java. You can also use Groovy or Scala. You can also use solutions like JRuby (Ruby on the JVM), Jython (Python on the JVM), Nashorn (JavaScript on the JVM). Further, the IDE support for a language like Java is utterly fantastic because of the static nature of the language. That’s not something you’ll get as well with a Ruby or Python IDE.

Automation Relies on the Future

Also keep in mind the future. As your suite of tests/checks grow you will find the need to add functionality to make everything easier to use, run faster, run in parallel, run on various grids, provide better reporting, run as part of virtualized containers, and so on. When this happens you will effectively be doing two jobs: (1) building, refining and maintaining a test framework and (2) writing automated checks. If you add components like BDD into the mix, you may be placing BDD-style solutions in front of your automation, which adds yet another layer of complexity.

So the bottom line here is that teams should choose a language that will allow them to do what they need and want to in a way that is most effective and efficient. I had some further thoughts on this topic when I talked about automation as a technique, not testing. My focus there was more on how automation is conceptualized but that post does stray into the area of how automation is implemented as well, with some focus on language choices.

Overall, I encourage testers to fight the idea that automation programming must be done in the same language as the development programming. There may be good reasons for that to be the case. But there are plenty of good reasons why that doesn’t have to be the case.

2 thoughts on “Automation Language Is Not (Necessarily) Your Development Language”

Jim Hazen says:

27 December 2015 at 6:55 pm

Jeff,

Excellent job on this post. I couldn’t agree more with what you are saying and emphasizing. This is a sane voice of reason type article, and I’ll be sure to pass it along to colleagues and reference it in presentations I do. I especially like how you laid it out on Selenium Webdriver, not that I am deriding it but it really is only and API like you said and not the end-all solution some people misbelieve it to be. Just like some of the commercial tools I use are not end-all solutions but merely tools that are used to get to a workable solution. All-in-all the real solution comes from the computer between your own ears.

Reply
Lisa Crispin says:

3 January 2016 at 5:07 pm

You make some excellent points, such as thinking about the future – our automated tests should be understandable and useful for many years. And I agree that who “owns” the tests plays a big part in what language to choose for automating them.

I’ve spent many years as a tester owning the automated tests (or checks or whatever you prefer to call them). I did a pretty good job, too. Automated UI regression tests I wrote in my first year at my last job were still serving as a good safety net eight years later.

Nevertheless, I finally conceded to the fact that, given that I could only devote 20% of my time to automating tests, I was never going to be as good at coding as my programmer teammates who spend 100% of their time automating tests. While it CAN work to have testers own automation, IME it works better when the TEAM owns automation. The testers do what they do well – specify useful test cases. The coders do what they do well – write robust, maintainable code.

When I joined my current team over three years ago, it was disappointing to me that I generally do no test automation. However, we are finding that not only are the programmers good at TDD, they are learning how to do BDD effectively. They often ask for my input on the tests, and overall we have a much better shared understanding of each story’s desired and undesired behavior. I now have time for much more exploratory testing at a feature level, discovering issues and missing stuff that I wouldn’t find testing each story individually, or automating the tests.

Every team is different, and I know plenty of testers who are superb coders. So I do think your approach can work fine. All too often, though, I see teams leave that pesky test automation to the “QA team”. As a result, not only are there no useful automated regression tests, there is also nobody with time for exploratory testing or testing for other quality attributes.

I’m nit-picking though, I must say your blog has been extremely helpful to our team as we learn good ways to do BDD! Thank you!

Reply

Stories from a Software Tester

Twice upon a time, in another space, no distance in any direction from here …