Actually, testing games is not all that much harder than testing other applications. What’s hard is sometimes being able to figure out how to test them well, given their numerous interfaces and means of interaction. Even harder is sometimes isolating the very specific, but very odd bugs that come up. I recently came across just this issue with a particular bug I found while testing a popular game.

In some of my freelance time, I do testing for game companies. I recently came across a bug that was, to me, really interesting when testing the game Star Wars: The Old Republic. The bug I refer to was found entirely by chance. I certainly wasn’t looking for this bug or for even any bug that could be considered related to this. When encountered, it was not clear at all what the actual issue was. Sounds like the story for a lot of bugs we find, right? But let’s consider some details.

For those who don’t know, Star Wars: The Old Republic (SWTOR) is a massively multiplayer online (MMO) game wherein you create a “toon” and run them around the Star Wars universe, taking on quests. These quests are usually presented to you in cut scenes where you see your character chatting with non-player characters (NPCs). After the cut scene is done, you go about your business.

As in most MMO type games, you will fight groups of characters (called “mobs”) and, assuming you defeat them, you will be able to pick up loot, which can consist of gear (armor, boots, gloves, helmets, etc) as well as other items. Some of these items — like armor pieces or weapons — can be equipped on your character. Further, your character will have a “companion.” This is a game-controlled toon that goes along with you on your quests and helps you out a bit during battles. This companion can also be “geared up” — such as by giving them some of the loot that you find.

So here’s what happened…

So during one testing session, I was going through a “flashpoint.” A flashpoint is sort of an episodic mini-story that basically gives you an overall objective along with a series of mini-objectives. So I was playing through this, doing some cut scenes, fighting mobs, picking up loot and just generally saving the galaxy.

Near the end of the flashpoint, during one of the cut scenes, I noticed that all the sudden my screen just showed the last contents of the cut scene and it was sort of jittering, for lack of a better term. A battle immediately starts after the cut scene — but I couldn’t do anything! Yet the game wasn’t frozen. I could hear that my companion was in a battle for his life. Further, when pushing my “abilities” keys I could tell I was actually doing damage, even though I couldn’t see what was happening because the cut scene screen was blocking everything.

So the game was clearly playing behind the scenes and the game mechanics were working. I could even continue to play it, albeit not very effectively since I couldn’t see a darn thing. My only solution was to log out of the game, re-log in and — sure enough that “fixed” it. Glitch? Maybe. MMOs do have that sometimes. So I kept on playing the rest of the flashpoint and things went good until the last cut scene, which ends the flashpoint. Sure enough — right at the end of the cut scene, the jitter effect reappeared and I couldn’t do anything save use my “workaround” of logging out and back in.

Okay, so Houston — we have a problem.

Let’s Figure Out What’s Going On Here

Now it was time to isolate. And this is where life gets interesting because I had to really put on my tester thinking cap here and respond as I would in any other situation like this.

First of all, was this a known issue with the game? Turns out: no. Or, at least, no one had reported it.

So I had to gather my information: I was playing as a Jedi Consular class on the Republic faction, doing a flashpoint that is available to all members of the Republic faction. I appeared to have limited the issue to cut scenes and, further, it was only right at the end of the cut scene. But … was it just flashpoints that this happened in? Was it just that flashpoint? After all, there are many. I was doing one called “The Esseles” but there are dozens in the game.

So how to test that? Well, clearly the answer was try a different flashpoint and also try quests given outside of a flashpoint.

It turned out that a quest outside of a flashpoint exhibited the same issue. In fact, I was now finding that any quest I took showed this issue. That was interesting because this had not occurred at the start of the game for me. I had been playing for days with this toon, with no issue. Even the flashpoint itself started out okay.

Hmm. That could imply something happened in the flashpoint. But … what? Was my toon somehow corrupted? Well, that’s a harder issue to diagnose so I considered another possibility first, one rooted more in hardware. Specifically, was this a graphic card issue? In and of itself, clearly not since I was playing the game with the same card for months. But was it perhaps my drivers? Could my NVIDIA card have done one of its automatic updates that I always forget to turn off? Turns out: no. Doesn’t appear to have been that. Plus the game itself was basically working fine. It was only during cut scenes and only right at the end when the cut scene was about to transition back to actual game. Such specificity seems to rule out what would have to be a very selective graphic card issue.

Okay – was it a resource issue? Had I been playing too long and game resources (and cache) were maxed out? No, because that was easy to check: simply restart the game entirely, restart the computer entirely, etc. The problem still persisted.

How about a patch? MMOs routinely patch their games with new content and bug fixes. However, there was not a patch between the time I started playing the flashpoint and when I ended it. And, as I had observed, I was able to start the flashpoint without issue. The issue started to manifest during the flashpoint.

Okay, so all I knew at this point was that everything was working fine for me in my game, including in the flashpoint. But at some point during that flashpoint play through, this issue manifested. So then I got back to: was my toon corrupted somehow? Did one of “those things” just happen where some element of your game ends up getting hosed with little rhyme or reason? Again, though, this would have to be a very, very selective hosing: only interfering with cut scenes and only at the very end of the scene.

Another thought occurred, though, that might help me understand the likelihood of my toon being corrupt. Like all MMOs, SWTOR lets you group with other real players. So I wondered: if I was grouped with someone, and this happened, was it just me that saw the effect or did everyone in a group see it? Ah, as it turns out, everyone in my group saw it. Yet they only saw it when grouped with me. Thus it was definitely related to my toon, but it also meant that it wasn’t isolated to my toon since it could impact other players in a shared instance of the game.

Okay, then I started to think: what happens during a flashpoint? Well, I already said — you fight mobs, you pick up loot, sometimes you equip gear. Hmm. Could something I picked up have been causing the issue? Could something I equipped have been causing the issue? Well, consider this screenshot:

You can click that for a bigger image, which should open in a new tab. It’s a fairly large image so you can see the details.

That shows you my toon — Ka’sira, the Jedi Consular. (Yes, I tend to roll female toons. If you’re going to role play, go all the way I guess.) Notice all those ‘slots’ surrounding her picture in the dialog window, which is called a character sheet. Each of those items could have potentially been an issue, I suppose. And this is just what I have equipped. It doesn’t show you my inventory, which had many more things in it.

Here’s What It Ended Up Being

Now, to spare you suspense, I will say that the issue was not with my toon. It was actually with my toon’s companion! Yup. Check out this screenshot:

Here I’m calling out Qyzen Fess’s gloves in particular. (For those of you who aren’t Star Wars geeks, you might be curious to know that Qyzen is a member of a lizard-like species called Trandoshan.)

It turns out those gloves — which I had picked up and equipped on Qyzen during the flashpoint — were, for some reason, causing cut scenes to bug out at the end. That’s why the flashpoint cut scenes worked at first — because I hadn’t found and equipped the gloves yet. Qyzen was still wearing the gloves outside the flashpoint and that’s why any cut scenes were impacted.

Umm … really? Yeah. It really was that specific. But not only that, keeping my tester hat fully on, I now had to consider: was it just those gloves?

You’ll note from the screenshot that those gloves are called “Gamorrean Wraidskin Handgear”. Did that matter? Was it just those? No, as it turns out, this problem also happened when I equipped Qyzen with “Echani Wraidskin Handgear”. Hmm. So maybe anything named ‘Wraidskin Handgear’? After all, that was a commonality. Nope, because it also happened with “Prototype Echani Heavy Handguards”. Well, that’s just great. Is it any gloves? As it turns out, no. I was able to equip Qyzen with various other types of gloves where the problem did not occur.

One other question then had to be answered. Was this just on the Jedi Consular class that I was playing? Was it just with Qyzen as a companion or did other class companions have the issue as well? The first question there is quite important because the Consular actually has two types of advanced classes: either Sage or Shadow. Essentially a Consular gets to choose an advanced class, but the companion is the same in each advanced class. I was playing a Sage. So I had to try on a Shadow and, sure enough, the problem happened there as well.

So Qyzen was simply an issue. (Stupid Trandoshans.)

The Problem (for Testing) Expands

The second question (“was it just Qyzen or other companions as well”) was a bit more concerning. SWTOR has the following classes: Smuggler, Bounty Hunter, Sith Inquisitor, Sith Warrior, Jedi Knight, Jedi Consular, Republic Trooper and Imperial Agent. That’s eight classes and each class has companions. (Each class has is own set of advanced classes as well, but the companions are the same, regardless of the advanced class you choose.) That’s a lot of testing to figure out if this issue is wider than just the Jedi Consular and if it’s even particular to Qyzen. For example, if any companion of a Consular wore the gloves, would the problem happen?

The even bigger question: is this just the gloves? Could other gear also cause the issue? Is there footwear that might cause it? A type of weapon? A specific implant?

Keep in mind this is a serious issue from a game play perspective. It’s not obvious what’s happening or why which means that a gamer figuring out that the gloves of their companion are the issue is unlikely. Further, this didn’t just affect the player. It affected whomever they were grouped up with, so everyone’s experience was compromised in this situation, not just the player with the class that had the companion that had the bug. The players affected stood a good chance of losing all the progress they made on a flashpoint that could take upwards of an hour to complete. Further, it wasn’t even limited to just the flashpoint, which was an optional quest anyway and could be avoided. This impacted all quests, including those in the class story, which players are usually working to complete.

So what you can see here is that testing games like this is just as hard as testing any other application and the consequences can be just as dire, both for users and the reputation of the company. The permutations of behavior in games are massive. But the above situation may get you thinking: if you had to test a game like this, how would you approach it? How would you partition the elements you need to test? How would you construct paths of behavior that you want to execute? How could an issue like the one I describe here have any chance of being caught before it reaches the public, except perhaps by luck?

This is a perfect example of why testing is hard and a very good example of how issues that testers have to wrestle with can be quite difficult as they work to isolate specific conditions that lead to bugs.

Earlier I had briefly talked about enhancing testing skills by playing games. I believed that then and still do now. I think examples like this show the challenge of testing in an effective and efficient manner when you have a large problem space to consider. In fact, I believe in this so much that I actually wrote a game — called “Test Quest” — that I use during interviews to test candidates. But that’s a story for a different time — although not in a galaxy far, far away.

In the meantime, happy gaming … and testing!