Testers must have the ability to utilize exploratory testing as well as scripted testing. But exploratory testing is actually not the entirely free-form exercise that some people associate it with. So let’s talk about that.

In previous posts, I’ve talked about the craft of testing as well as the idea that there is a component of art to testing. I do believe that craft, and that art, are most on display in the context of effective, efficient, and elegant exploratory testing.

In many cases, exploratory testing requires you to combine test design with test execution, coupled with learning about the application itself. Exploratory testing is essentially the act of investigating a problem space that is often very large, very complex — usually both! — and that has a shifting, dynamic structure because the parts have interdependencies.

James Bach has already posted numerous thoughts about exploratory testing and his thinking is probably way in advance of mine. So I would urge you to read his exploratory testing posts. Whether I agree or disagree with James, he invariably makes me think. I’m going to take a more example-focused approach for this post.

I should add that exploratory testing is also where you can often most effectively “bump the lamp.” Earlier I talked about bumping the lamp while testing Grand Theft Auto 5. This notion also came up again recently in a game I had done some contract testing for, which is Star Wars: The Old Republic (SWTOR, for short). Since I have provided past examples of this work (see testing games is hard and gaming like a tester), here I’ll cover a specific example.

Exploratory Testing for User Immersion

Testing games is not just about testing functionality, it’s also about testing the space in which the game events take place. Part of this is consistency of appearance and another very large part of it is immersion. Narrative and situational immersion in a game is important to many players and it’s one of those things that many players don’t even notice is happening — until the immersion is broken. When this occurs it’s referred to as “mimesis-breaking” and exploratory testing is one of the most effective ways I’ve found to look for these kinds of issues.

In fact, I’ve found that exploratory testing guided by personas is particularly effective. I highly recommend James Whittaker’s book Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design as it gets heavily into this idea. While I’m describing this in a game context, the techniques work for any application or service.

So let’s consider one example that I came upon when testing the game. It was not something that I could initially have scripted for. It was something I discovered by exploration and, in fact, this came up when I was doing a presentation to the SWTOR developers about testing. I’ll show you a series of screenshots here to make the ideas clear. Click on each image if you want to see a much better resolution image.

In the scenes you’ll see here, I’m on this dust-pit of a world called Korriban. (For those of up on the new “Expanded Universe”, this world is now called Moraband.) I wandered into this tomb — Ajunta Pall’s tomb, if you’re curious — and I happened to be looking around and, in one room of the tomb, I looked up and saw a hole in the roof.

I could see a planet or moon through the hole and this made me stop for a second because I couldn’t remember where on the surface this hole could have possibly been. So I checked out my map to see where I currently was located:

What a surprise. It told me I was located exactly where I was. What I needed to see was where that location corresponded in the wider world. In the larger image from above, I circled that so you can see where I had to check. And here’s what I saw:

So you can see (by the circles) that my area view shows me in the tomb but now the world view is showing where I am in a wider context. I noticed that I was positionally over the “a” in the word “Dark” in the phrase “Valley of the Dark Lords”. (Not a vacation spot, let me assure you.) So I cruised up out of the tomb and went back to the area in question. You might notice here that ground level doesn’t seem to show much of a hole:

I made sure I was standing in the position that I should have seen it:

As you can see, I’m over the “a” in the word “Dark” in the phrase “Valley of the Dark Lords”. So I’m in the same spot that I was underground, but now I’m above ground. I found the lack of a hole disturbing. (Minor Darth Vader reference and joke there. Extra points if you got it.) Further, just to be sure, I took my camera view up so I could get a look and — yep, as I thought: no hole in the ground.

Now, would I have scripted a test for this? Probably not. I didn’t even know to consider it until I looked around. And even then, it only struck me because I was checking out the interior of the tomb and really looking around, and happened to correlate one observation here with another observation that I didn’t even really know I made. This inconsistency jumped out at me only because I couldn’t think of any place on the surface where that hole would have led to but notice that this required me to keep in mind the problem space I was dealing with. This, I believe, is an important part of exploratory testing. You have to keep a lot of things in the back of your mind and allow your intuition and experience to guide you a bit.

This was also a “bump the lamp” moment for me. I have no idea how many players would even notice this discrepancy and, of that group, I had no idea how many would even care. But I cared. I wanted it to be right. Or as right as possible. Part of exploratory testing, I believe, is also providing a basis for why you use exploration as a means of solid testing. In this context, what solutions could there be to this problem? The developers may say “Well, we’re not going to just change the map. And we can’t entirely remove the graphic of the shaft that leads to the surface.”

I might (and did) say: “Well, in my exploring in other areas of the tomb, I noticed some of the holes to the surface were covered over like this:”

The point here is I argued to the developers that a ready-made solution was already in place. We could just do a render of that particular shaft as closed rather than open.

Immersion But Also Leading to Functional Issues

Exploratory testing also led me to notice this:

In the larger screen shot, you’ll see that my character appears to be levitating. (Yes, she’s a Sith but she’s not that good yet. Levitation is strictly out of her league at the moment. She’s great with a lightsaber, though.) In that scene, notice I’m near a step there? Let’s try moving more onto the platform:

Ah, yes. So there my feet look well planted on the ground. Okay, so maybe we have a layering bug here. Let’s step fully off the platform and steps:

Hmm. I’m still kind of levitating there. Okay’s move even a little further from the steps:

And I’m back on the ground. So that’s an issue I need to bring up. It’s a little immersion breaking. Now I’ll go to that green doorway:

Whoops! I’m levitating again! But here I’m pretty far from the steps. In fact, I’m about as far as I was previously (when my feet were on the ground), just in a different direction.

So by exploring, I’ve clearly found that the layering in this room is off. Oh, and by the way, keep in mind that you control your camera in this game. So as a tester it could be very easy to miss these things because normally your camera is above and behind you, sort of looking down on you from the back. This means it can be difficult to see if your feet are actually touching the ground. You have to rotate your camera around to see this.

Exercise: There’s another thing to be watchful of in these games when you have a character moving through the outdoors and indoors like this. Can you spot what that might be?

Incidentally, this particular issue that I’m bringing up here can be immersion breaking to an extent. But it has even bigger ramifications. By exploring this I was able to realize that the game rooms were, in many cases, not adequately representing height differences between elements of the room. That can be a huge problem in a game, like this one, that has so-called “jumping puzzles.” I won’t get too much into this here but basically there are these things you can find in the game called “datacrons” and in many cases you have to leap on pipes, boxes, across gaps, etc. This can be very frustrating when the game is not layering the environment correctly because what looks to you like a correct jump leads to you falling.

There are plenty of YouTube videos on datacron hunting if you are curious to see what some of this looks like in practice.

So here my exploration led me to realize that previously vague bug reports (like “jumping puzzles are too hard”) could be quantified a bit for looking at variances in how the in-game character was represented in the environment along the x, y and z axes.

Explore a Model

Now let’s step back from the game concept for a bit and just do a simple exercise. Consider the following screenshot:

Just by exploring that static view, what comes to your mind? Is anything triggering alarm bells? How would you explore this particular small bit of functionality were it live in front of you?

One of the first things testers tend to notice is how the quantity field appears to be either a drop down or a spinner or perhaps both. (In fact, in the application it is both. The field provides the up/down buttons but it also has a dropdown.) That immediately should ring some alarm bells. What if there are hundreds of items? Forget hundreds — What if there’s even just 50? So the drop down contain 50 entries? Seems kind of silly, right?

Certainly another thing that sticks out is the “Add to Cart” and what should be in a testers mind is that this button should not work if there is no quantity of the item available. A message should presumably be displayed in that case. In fact, perhaps the “Add to Cart button” should not even display at all if there is no available quantity. The quantity field itself should probably show 0 (and be disabled) if there is no quantity of items available.

Now if I make that quantity field a text entry field, then I need to make sure that whatever quantity I enter is not greater than what is available. But how will I even know? So there would need to be some indication of the total quantity available and then a means to recognize and respond to attempts to go beyond that availability.

Further, I would expect some message should appear automatically if a product has no quantity. I, as a user, shouldn’t have to notice the 0 in a field or the fact that the Add to Cart button is grayed out or not there.

Also, are there “Tech Specs” on any product? I’m looking at a bottle case there. Would there be “tech” specs for that? Maybe. But in exploration, I’m questioning what that would mean. Perhaps it means dimensions of the bottle or the full case itself. Or maybe it indicates liquid carrying capacity. But just as an exploration exercise, that stuck out at me.

Now here’s where I’ll tell you that this is from a real application. And, in fact, most of what I indicated I would expect via my exploration was not the case. There were no messages when the quantity was 0. But, in fact, as it turns out, the entire bit of functionality was not connected to the inventory subsystem at all. So there was no way to know the actual quantity available which made for an interesting test exercise when you think about it. Or, rather, it led to an interesting business discussion (and thus a testing-as-design activity).

What to take from this, however, is that just by exploring something by sight, you are able to think about a test approach and a series of scenarios you would like to try. I’m sure, just looking at that image, there are various other things you can think of to try.

Explore a Problem Space

Now let’s give you an actual space to explore. I’ve been constructing a small web site and application for use on putting my automated tool solutions through their paces. One of these sites is called Decohere and I have a Project Overlord portion you can take a look at.

Without scripting (and without access to any requirements), just explore that part of the application. See what you discover.

There’s one particular bug that may not be obvious and most testers don’t think to script for. It’s a hidden bug, to a certain extent, and, earlier, I wrote up the idea of hidden bug analysis. I specifically put this bug in because I use Project Overlord as a way to test people on test writing skills. In other words, would your tests have found this bug once I introduced it? In my experience this particular bug is found more by the explorers than by the scripters.

Always Explore!

Attention to detail is a large part of being exploratory. I think my game example here showed that a bit. It’s not just a matter of seeing what’s right in front of you but it’s a matter of remembering what was right in front of you earlier and correlating that with what you are seeing now. In the realm of testing, this is sometimes described as using “consistency heuristics.” That’s part of it, to be sure. But there’s quite a bit more to it than that.

Being exploratory is a lot like having a particular sort of muscle. You have to exercise it fairly regularly and in a variety of different domains. Exploration is about keeping a model of something in your mind and then exploring the real thing while updating your mental model. The focus is on investigation and curiosity, coupled with the ability to learn new information and adapt to changes in information. It requires the ability to change modes, from instrumentalist, to operationalist, to theorist — pretty much on the fly.

A lot of testers say “Well, duh. I do all that already.” Yet I’ve found many, many testers are not very good at this. Or at least nowhere near as good as they think. (The same no doubt applies to me too, lest it be thought I’m painting with a very selective brush.) Part of my own exploring about how testers think and observe, and thus explore, is what led to me creating my game Test Quest. (I’m still not entirely sure whether I should be embarrassed by that or not.)

Just note that exploring does not mean giving up on test writing or on any sort of scripted testing at all. Exploratory testing is an approach: one of many. But it is an approach that is one component of allowing testing to be effective, efficient and even elegant as well as an approach that is key to the idea of tester as learner. Finally, I firmly believe that a focus on exploratory testing hones your skills to be a better storyteller.