Here I’ll go back to a game I talked about previously and show some interesting game bugs, all of which came out of exploration and where the finding of one bug guided exploration to finding others, which led to some causal mapping. Of course, the idea of “bug chaining” and “causal mapping” is certainly valid in any context, not just games. But games can certainly make it a bit more fun!

So the game in question here is Elden Ring and this was a game I last talked about in the context of ludonarrative testing. So let me give you some context here in case you’re not familiar with the game When you start up a new game of Elden Ring, your character will immediately run into a boss character called the Grafted Scion.

Don’t worry. He’s incredibly friendly. Really.

Actually, no. He’ll pretty much kill you outright. And, in fact, that’s the goal of this boss. At the start of a new game (and assuming you aren’t in what’s called New Game Plus or NG+), you really don’t have the stamina, the hit points, the focus points, or the armor in order to engage this guy all that well. You are meant to die to him. It’s way of acclimating players to the idea that you’re going to die. And do so often.

However, all this being said, you can beat him. You have to be pretty good at the dynamics of the character you’re playing but it is certainly possible.

Bug? Exploiting the Game

With that bit of context understood, early on there was an issue where you could gain invincibility (without a cheat!) and beat the Grafted Scion with no skills whatsoever. Check out the video first. When you watch this, you’ll see that I let myself get killed to the creature right away. My red health bar (at the top left) immediately depletes and I’m dead.

What you can’t easily see is that as I died, I hit ALT+F4 to get out of the game and shut it down immediately. Use case: I’m a gamer that just rage quitted because I died. (Extra points if, as a tester, you thought: “ALT+F4. That’s PC. What if I’m playing on a console like Playstation or X Box.”)

What the video shows you, however, is that when I restart the game, I can go back in to fight the Grafted Scion and — well, what’s happening there? Watching the video, what are you seeing? As a tester, what are you observing about this bug that I just submitted. Imagine this video was attached to my bug report.

You’ll probably notice my health (again, red bar at the top) is entirely depleted. You’ll also likely notice my character is clearly taking hits from the Grafted Scion as she gets walloped a few times. Yet, I don’t die. So I could essentially take my sweet time and kill the monster. All with no cheats or anything.

Thus a bug: quitting the game after death apparently has some impact on the fight you were just in.

Of course, this would lead to a lot of questions and thus possible experiments: is that only for this Grafted Scion character? Is it only for boss characters in general or would it apply to any character in the game? Is there some time limit where it’s “too late” to quit the game? Do I even have to die to get this same effect?

I should note that some people refer to the above technique, if such it an be called, as “cheesing” the boss character although that’s not really true. “Cheesing” in a game context tends to mean that you do something that is very easy, does a whole lot of damage to something, and requires little to no skill on your part. But key to that is that you use in-game mechanics or dynamics to do this.

Here the bug was about an out-of-game mechanic (i.e., I quit the game) so this is really more of an exploit.

I should note that this ability to quit and gain invincibility was patched out of the game. For those curious, why this worked originally was because FromSoftware’s engine uses triggers to determine the state of the world. There are local triggers and world triggers. In the above case, depending on exactly when you did the ALT+F4, the local trigger said you died but the world trigger wasn’t updated yet.

So when you came back into the game, your health was “stuck” at the lowest health you could be just as you died. Yet since the world trigger was not updated, no hits by the Grafted Scion could harm you since as far as the game was concerned (local trigger), you had no health to deplete. Thus you were effectively invincible. So, in effect, you had a bit of a race condition here.

Bug? Cheesing the Game

Now let’s consider an example of actually cheesing the Grafted Scion. Check out the next video.

For this, you are essentially using some trees to get yourself into a spot where the creature effectively can’t reach you.

This too was mostly patched out of the game. I say “mostly” because the geometry was made a little trickier in terms of being able to jump on the tree limbs and, further, the Grafted Scion was tweaked to be a bit more aggressive, which includes his ability to issue a scream that he can aim “upwards” into the trees where you’re standing.

Also worth noting, perhaps, is that even if this cheese is used, it required a character like the Samurai you see in that video. The reason for this is that you needed someone with a ranged weapon and enough ammo for that ranged weapon. If you were a melee character, you would not be able to remain far enough away from the Grafted Scion. Even a ranged character like the Astrologer (which is the game’s version of a sorcerer) doesn’t work because the character doesn’t have enough magic resource to take out the creature.

So … Bugs?

Now here’s a question: are both of these bugs? I labeled each section as “Bug” with a question mark for a reason.

Well, let’s think about this. You could argue the first (the exploit-quit) is a bug since it’s clearly using an exploit of a mechanic that is out-of-game. A developer could certainly say: “Well, so what? So the player gets around the boss.” Okay, but if this exploit can be used here, perhaps it can be used with some of the other boss characters in the game, essentially making the game trivially easy when it’s supposed to be at its hardest.

Certainly more seriously perhaps the exploit could be used during PvP sessions with other players, which would give one player an obvious and complete advantage over the other. And, of course, if both players did this, PvP would become useless. Why bother fighting if you and the other player are both invincible?

The second technique (cheese-from-tree) could be considered a bug if it’s the case that you are not supposed to be able to kill this initial boss at all. Yet, as I mentioned, you can do this even if you aren’t cheesing it. It’s hard, but it’s entirely doable.

Unlike in the first instance, here the player just used some geometry to give themselves an advantage. Now, granted, the trees were not meant to be leaped to in this way for the area. That said, from an in-world perspective, it certainly makes sense that a player could do it. And figuring out how to leverage in-world geometry to gain yourself a tactical advantage doesn’t seem like a bad thing. In fact, in some encounters, it’s entirely necessary, such as using objects in the world to cause enemies to lose their line of sight to you or to block some of their attacks.

Chained Bugs

The above bugs were chained in that they were kind of related. One of the test exploration goals was to find a way to defeat the initial boss even if I didn’t have the pure combat skills to do so. The one finding (exploit-by-quit) was found by acting as a player who got ticked off. It was a bit fortuitous of a finding although it also stemmed from other exploratory testing going on: what happens to the saved game state when the player “hard quits” the game?

The second bug was found as a result of the first bug being patched and thus looking for other ways, beyond the planned “straight combat,” to defeat the initial boss.

All of this actually led to discovering another bug when you combine the two issues just described. The idea here is you go to fight the boss but don’t die. Just get in there, wait for him to engage you, and then ALT+F4 out of the game. If you do this, you’ll be brought back outside the initial gate where you meet the boss. But since you didn’t die to him, the previous bug issue (which was resolved) is not operative.

However, the gate to get to the boss is now closed off to you with a yellow cloud barrier. This is because when you engage a boss, the game will usually bar your exit from the area. You can’t leave until either you or the boss dies in combat. Once the boss is dead, the yellow cloud barriers are removed and you can move on. But, in the test just described, you “left” by hard quitting and, given how the game engine works, it will never deposit you back into a boss arena. So it leaves you right outside of it.

So now take a look at this video. Having little to no armor on helps with this, although being green is totally optional!

As the video shows, if you now find a way to get back into the gate– but without crossing the gate (using a variant on the cheese-by-tree) — the boss is essentially static. You can see in the video that the Grafted Scion isn’t moving and, even if attacked, it doesn’t engage. Thus it can’t be killed. Also the yellow cloud barriers remain. They remain because the boss isn’t dead but since you can’t kill the boss, they will now never disappear. Thus you have to find a way around them. And using the cheese technique, you can do that here, which the video shows.

Yet this is showing a much more serious issue: bosses can become bugged to where they don’t engage at all. And that can stop story progression in some cases, since defeating the bosses is a large part of how the story moves forward.

Using an Explore-Exploit Test Technique

Notice how all of this was exploration of the mechanics and dynamics of the game. This was, in effect, using an explore-exploit dynamic. And here by “exploit” I don’t mean the same thing as a game exploit. I’m referring here instead to the explore-exploit tradeoff.

In this context, once I, acting as a tester, explored certain elements to get something to trigger, I then exploited that mechanic or dynamic to guide further exploration. Here “explorations” means finding out what you don’t know and “exploiting” means leveraging what you already know.

Also worth noting: no amount of automation in the world would find things like this in any sort of reasonable time frame. Even if you were applying machine learning algorithms to this via a test orchestrator, the level of training and model building it would have to do would be prohibitive. (Keep in mind my simple Flappy Tester example, based on using AI with Flappy Bird, that I used in my post Testing AI … Before Testing With AI.)

There is one other thing to understand about FromSoftware games: their jumping and platforming mechanics are routinely pretty bad. Their geometry does not line up very well on their design engine, which is why they have to patch it so many times to remove abilities to cheese or to remove things that allow players to get to areas they shouldn’t be able to. Further, in Elden Ring it’s the case that you, as a character, can jump. Later, you get a horse. And that horse can jump. Not only that, but the horse can do what’s called a “double jump.” Thus the horse can allow you to get into areas or around barriers that the character simply couldn’t do if not riding the horse.

But do you do that everywhere in the game world? After all, Elden Ring is a huge open world game. Well, clearly you apply the testing where it most matters. Since we discovered potentially game breaking behavior with story-progressing boss encounters, performing the explore-exploit loop at all boss encounter areas seems like a worthwhile way to focus testing.

Exploring, Chaining and Causality

What all of this shows you, as a tester, is how much you have to explore and find ways that you can exploit or cheese the mechanics or dynamics of the game. These can lead to bugs that may be helpful to the player (but, as a game designer, you don’t want them in there) to bugs that are outright detrimental to the player. Sometimes such bugs can be linked together in an interesting chain. And you can bet that if, as a tester, you found this stuff, users will do likewise.

That chain ultimately tells you about weak spots in your overall design, such as, in this case, how to place trees or cliffs or where to situate boss characters that are supposed to be gated by uncrossable barriers and so on.

I would argue, with a great deal of evidence to back me up, that doing all this in game testing is so much harder — and thus requires so much more skillful testing work — than it does for non-gaming applications. That said, of course, non-games can be very complicated in terms of their interactions and so the idea of exploration and bug chaining is relevant regardless of the application you find yourself in.

Finally, what this really gets into is creating a causal model around the testing you do. A causal model, as you may guess, requires considering causal relationships. And in looking at such relationships, you need a language of knowledge (what we know) and you need a language of queries (what we want to know). Cause-effect forces operate in any environment we provide to users as part of an experience.

A causal map provides a way of understanding the cause-effect relationships that exist in what you are exploring. You have to go up a ladder from association (which is seeing or observing), intervention (doing or intervening), and counterfactuals (imagining and retrospecting). One book I feel all specialist testers should be acquainted with is The Book of Why by Judea Pearl and Dana Mackenzie. In that book you’ll find a Ladder of Causation:

We often test to gather what the experience of something will be and determine if there are risks to that experience. This manifests, often, as just data. Maybe some test cases at the start and some test report at the end. But data is opaque to cause and effects. Data by itself doesn’t guide us when there are causal questions in play. Think about trying to write up the above explore-exploit techniques as a set of test cases. You probably could. But would that really be helpful? Would a simple test report saying “Boss can be cheesed by jumping on a tree” be as useful without all of the surrounding context that led to it and came out of it?

Causal models are a topic for another day but, for now, the groundwork has been laid: exploring, exploiting and chaining. Those are the techniques that testers can use to start applying grids of causality to experiences that are discovered through specialized and skillful testing.