Previously I talked about testing like a gamer. So now let’s talk about the other side of that.

A lot of people I talk with think that being a game tester is all fun and … well, games. But it’s not. It actually takes a lot of discipline to approach a game from a critical standpoint that goes beyond “I like it” or “I don’t like it”. Being able to spot bugs in games forces you to engage numerous aspects of how your brain takes in and processes information into a cohesive narrative of experience. This experience is, in my view, one that all testers should go through and experiment with. In fact, this is one of the reasons why I wrote my challenge game Test Quest.

In this article I want to share some examples of the things you have to be watchful of in a game. Then I want to give you a few fun challenges of a game-related nature where I’ll ask you to spot some bugs in a few browser-based games. My point here is not just to appeal to those who enjoy gaming but to provide a certain perspective about testing as a discipline.

Finding Game Bugs Requires Diligence

I’ve done some official contract-based testing for Bioware’s Star Wars: The Old Republic and I showed an example of one bug that was difficult to diagnose (see: Testing Games is Hard). Here I want to drill that point home by showing some other aspects of testing on that same game.

For any of the pictures, click on them and you’ll get a nice screenshot of the game in a different tab with some annotations that I hope make each respective issue a bit clearer.

Object Placement

Let’s talk about one of the more tedious aspects of game testing: finding things that, quite frankly, look stupid but aren’t game breaking in and of themselves. Here’s an example of what’s sometimes called a placement bug:

Either that guy’s chair is really uncomfortable or we have a rendering bug. To give you an idea of what “good” looks like:

Actually, another interesting thing — that you can’t tell from the static picture — is that the woman sitting in the chair is animated and is clearly meant to be talking to someone but, as you can see, there’s no one there. (Perhaps she’s just using her Bluetooth ear device. I’m sure they have those in Star Wars.)

Now, you may say: “By the Gaming Gods, Jeff! Why would you start off with such trivial nonsense ?!?” Well, here’s the interesting thing. In that same area, there’s another rendering bug:

In the full screenshot, you’ll see that I called out where the previous guy with the chair problem is.

So we have an interesting set of situations where we might have a room overlay that is not working. That can impact character movement, including your own player character movement. If you’re not a gamer, the “player character” refers to the player you control. In the screenshots, you are seeing my player character Ky’zen (along with his faithful, if sometimes irritating, droid T7).

These rendering problems are one of the leading causes of so-called “stuck” bugs in games like this. This is where you can move characters into areas that you are not supposed to and get them stuck because the “priority lines” that make up the drawing of the room do not map correctly to the “control lines.” Lots of game design jargon there and I would say don’t worry too much about it. My point is simply this: these problems, of which I’ve found two in the same are, are now indicative of a potentially bigger issue than just a few non player-characters who seem to defy gravity.

Here’s another example of where this can get bad:

Now, granted, I’m a pretty cool Jedi so perhaps I just kill those bad guys so quickly their bodies literally don’t even time to fall to the floor. But, more likely, we have more issues here. And in this case the issue can manifest as enemies who are able to stand in areas that you can’t reach or that you can’t impact with your weapons. In the worst cases, this means the enemy can keep attacking you, but you can’t attack them back. That can stop progression depending on the circumstances.

There’s a wider point here which is that it not only takes a certain amount of diligence to look for these issues but also looking for the possibility of whether or not a few of the issues are occurring in a similar enough context (such as an instanced game area) that may be indicative of a wider problem within that area.

Issues like this one also can indicate issues where the “floor” of the particular area you are in is not being rendered accurately. That can have effects that range from visually stupid, to moderately annoying movement, to completely unable to get to a particular spot.

Progression Indicators

In games like this, there are often “mission givers” or “quest givers”. And there’s usually an icon that displays above the mission givers head to let you know that this is a particular character worth talking to. Let’s consider what it looks like when it’s working:

In that screenshot, you can see that I’m facing a particular mission giver and it’s visually indicated as such. If you look down in the lower right corner, you’ll see the game’s “mini map” and I’ve circled how that visual indicator appears in your map as well.

Now here are some bugs:

In both cases, you can see that the mission giver icon does appear on the mini-map (lower right corner) but the icon very clearly does not appear above their heads.

Now, here’s a question for you testers: can you spot the potential bias effect that would occur if you are a game tester?

It’s simply this: once you start to know where these mission givers are and you’ve tested this level for the six millionth time, it’s pretty easy to overlook this kind of thing. I’ve watched testers blissfully regression test particular quest chains, each time failing to notice that there was not a quest icon over the head of the mission giver. And that’s simply because they knew that this was the mission giver. This is no different in principle to when you have testers who selectively ignore JavaScript issues in a web page when they know “that’s just how JavaScript is.”

You also see some people with the mentality of “Well, the icon does show up on their map. So the players should just keep on eye on that. How hard can that be?” This is akin to testers and developers feeling that users should be “smart enough” to figure out the interface they built, regardless of how unintuitive it may be.

Mission Progression

So now we get into something that gets a little more game breaking. Consider this one:

You can see the text of the mission (upper right corner) says you should click “override the elevator.” That’s the big keyboard looking thing to the right in front of my character. The reason the mission is structured this way is that it forces you to see a bit of dialogue before you use the elevator. That bit of dialogue provides a context for a choice you must make later in this particular mission. So the idea is that the elevator cannot be accessed (the blue button on the left, in front of my character) until you have done that.

Except what the screenshot above is showing is that if you just click the leftmost elevator button, the game will let you do it — it’s highlighted blue and thus capable of being triggered — and you’ll be able to skip out on a particular story choice that does matter. This skipping can, but is not guaranteed, to bug out the final mission, negating what would probably be about forty-five minutes of questing.

So the fix required is simply to make sure that the player can’t skip out on this. This particular case I’m showing here is particularly bad because by this point in the game, players will have become very used to clicking those dark blue elevator buttons and may entirely miss clicking the keyboard next to it.

This next issue is interesting as well:

Notice the mission text (upper right corner) says that I need to obtain a clinic passkey. When I do so, I should be able to click on the terminal (to the right of the character) and enter the area. I have my character’s inventory open and you can see I have a clinic passkey. However, the terminal is not active. It’s hard to tell that in the picture but it would be a brighter blue.

But check this out:

You can see I have three passkeys now and the terminal appears to be active. But here’s what’s interesting. The terminal wasn’t active when I had two passkeys. I had to have three. But — and this is an odd thing — it was the second clinic passkey that was actually working. You can kind of see that the second key is highlighted a bit in the inventory. So the issue here is that I had to have three passkeys but it was the second of those passkeys would trigger the effect. Meanwhile, the mission text indicated that I only needed one passkey.

I could have just written this up as “Unable to access Face Merchants mission area when clinic passkey is in inventory.” However, this extra investigation on my part, in terms of getting multiple passkeys, made for an interesting question: why can I even get three clinic passkeys at all? If the idea is to get just one, then why does the game allow me to get more than one?

You could argue: “Well, who cares?” Well, understand that in these games obtaining these kinds of “mission items” usually comes about during what’s called a “loot drop.” I need a clinic passkey so likely there are mobs of enemies I have to fight and in doing so, one of them will drop a clinic passkey. Once the clinic passkey is dropped and I collect it, there is no need for other mobs to drop such a clinic passkey.

Clearly, however, mobs were dropping multiple passkeys when I tested this as evidenced by the fact that I had three of them. I won’t bore you with too many more details except to say that this investigation prompted some checking of the code and it revealed that in fact the code of the game was originally set up to require three clinic passkeys. That bit of code was taken out, requiring only one. But a side effect of the code removal was that three passkeys were still required and the second was the one that actually was used.

But … and here’s the real kicker … I found that this wasn’t always the case. Sometimes in my testing, I found that I could collect just one clinic passkey and the terminal would be active. However, I noticed that when this “only needs one passkey” condition occurred, there were no other players (simulated or otherwise) in this particular area doing this same mission. Ah ha!

So here’s the trick: when multiple players are in the same area, they are killing the same mobs as you to get those passkeys. So the game has to “respawn” these mobs so that multiple players have a chance of completing the mission. When there were “enough” players in the area, the game code was reverting to requiring three passkeys. When there were “not enough” players in the area, the game code was requiring one passkey. Here the “enough” and “not enough” had to do with specific code in the game engine that kept track of loot drops, number of players, and respawn rates in this particular area.

It’s a little more involved than I’m making it sound, but not much. The point here is that just writing up this initial finding would have likely been dismissed by the developers. They would have tried to recreate the issue on their development shard and likely found that the problem wasn’t there because it was just them on the instance playing through the mission. And, trust me, the developers don’t sit there and play missions each time a bug is written. They tend to look at the top level code that controls the missions. So finding issues like this requires you to investigate but also understand the context of the game engine in which various events take place.

Information and Immersion

I mentioned earlier that these games rely on quests or missions. Those are often presented with a bit of text to situate you in terms of what you have to do. Those details can matter to greater and lesser degrees. Consider this:

Notice how the mission text mentions a “Duros named Deringon Lobacc.” Well, that guy just to right of the mission log — who is Deringon Lobacc — sure ain’t a Duros.

But here’s kind of an important point that’s easy to let slide. You have to care about these details but you also have to be able to know the domain of your app (game or not) to recognize inconsistencies like this. For all someone not versed in Star Wars knows, “Duros” could refer to a gang or some type of human or whatever. But as a tester I need to make sure I know the species of this universe so that I recognize right away that the text is incorrect.

Oh, by the way, you might notice that the character also has a problem of no mission icon over him even though one shows up in the map!

Lots o’ Testing

For any one of these issues that I showed you here, consider that this particular game has eight classes and those classes have hundreds of potential mission chains. From a testing perspective, you can manage some of this by realizing that some mission chains are shared between classes. Still, you are talking about a whole lot of testing across a very broad swath of functionality. Consider also that some of the mission chains have varying ways that they can play out and you can see the magnitude of the testing effort.

It’s a good exercise for testers to consider how they would go about testing this kind of game.

Spot the Possible Bugs

So now let’s take a look at two fun examples. Both of the games here are variations on the traditional Space Invaders game. There’s no “trick” to the games. Both of the (possible) bugs here are based on observation. It may take you a few playthroughs.

Alien Test Invaders

For this game, the goal is essentially to not get killed by aliens coming down the screen at you. Those aliens will be shooting and you can be one-shotted. That’s about all I’m going to give you except to say that keep in mind with any game, it should be required for the player to at least do something in order to win.

Play Alien Test Invaders (opens in a new tab)

Space Test Invaders

This game is very similar to the previous. Although here the aliens aren’t marching down the screen at you. But they are shooting. And their aim is dramatically better and each hit lowers your health.

Play Space Test Invaders (opens in a new tab)

Testing and Gaming is Fun

I hope this article gave you a glimpse into the scope of game testing. I also hope that I was able to demonstrate what is a truism for any sort of testing: it’s a discipline based upon skillful observation. A key part of this observation is the ability to maintain focus and rely somewhat on intuition. It also relies on your ability to spot patterns (both possible, probable, and actual) and determine to what extent those patterns can be exercised to determine if quality suffers in any way.