Testing Like a Maniac

So far in this series, I’ve tested Adventure Uncatridged, Gauntlet Reforged, Improbable Mission, and Laser Vault. This was all fine but I think it’s time to act like a maniac.

In 1987, Lucasfilm Games released Maniac Mansion. A meteor has landed near the Edison family’s mansion, and twenty years later Dr. Fred, Nurse Edna and Weird Ed are all under its control. Dr. Fred has kidnapped Sandy, the girlfriend of a kid named Dave, for an experiment involving her brain. Dave and two friends go in after her. You point and click: pick a verb, pick a thing, and a sentence like “Unlock front door with key” builds up at the bottom of the screen.

If that sounds ordinary, it’s because this game is the one that made it ordinary.

It’s also a game that took its own design seriously. You choose two of six other kids to go with Dave, and each kid has skills the others don’t. Bernard can fix things. Syd and Razor are musicians. Wendy is a writer. Michael is a photographer.

Depending on who you brought, the game has several different ways to win, and, of course, several different ways to make sure you never will. The house has its own schedule, too: Nurse Edna leaves the kitchen, Weird Ed gets hungry, a package arrives and the doorbell rings, all whether you’re ready for it or not.

To build all of that, Ron Gilbert and Aric Wilmunder wrote a script language and an engine to run it. They called it the Script Creation Utility for Maniac Mansion: SCUMM. It went on to run Zak McKracken, Indiana Jones, Monkey Island, Day of the Tentacle and more. And a big reason any of those still play today on modern machines is ScummVM, which began as a reimplementation of SCUMM. It started with the later versions of the engine and worked its way back to this one.

I really enjoyed this game when younger, so as a sort of tribute, I built Maniacal, a reimplementation of the SCUMM interpreter for Maniac Mansion, in C#. It’s a port of the parts of ScummVM’s code that this one game uses. It’s open sourced, and it’s on NuGet, so if you have .NET 10 or later you can install it with a single command:


dotnet tool install -g Maniacal
maniacal 

It plays the English releases of the game. There are the three this post is mostly about: the original Commodore 64 version from 1987 (read straight from its disk images), the first DOS version, also from 1987, and the enhanced DOS version from 1988, with its redrawn graphics. Later came the NES cartridge from 1990, the Atari ST and Amiga versions from 1988, and three demos, two for DOS and one for the C64, which brings it to nine. Like Laser Vault, it isn’t playable in a browser, and you need your own copy of the game.

Maniacal ships no game data at all: no scripts, no pictures, no sound. If you don’t have a copy lying around from 1987, I have made these available as Maniac Mansion Classic Preservation on my Archive account.

One interesting part about this is a comparison to Laser Vault, in that with the laser disc games, the original wasn’t something I compared against but the program I ran. This time the original is the program I run and there’s somebody else’s machine that runs it too. That turns out to be a mixed blessing, as you’ll see.

Let’s talk about testing it.

What’s Different This Time

The Laser Vault post drew a line I want to keep using. On one side are the games I wrote, where the original (if there was one) was a reference. On the other is the emulator, where the game is the original program and what I wrote is the machine it runs on.

Maniacal is on the emulator side of that line. The game itself, everything that makes it Maniac Mansion, is a program written in SCUMM: hundreds of scripts that say what happens when you open the fridge, when Edna sees you, when the doorbell rings. Those scripts come from the original game files and run unchanged. What I wrote is the interpreter that runs them: the script machine, the rooms, the walking, the costumes, the verbs, the sound, the saves.

But there are two things here that Laser Vault didn’t have.

  • A second machine. ScummVM runs the same game files. So I can do something a tester rarely gets to do: put the same moment of the same game on two independent implementations, side by side, and compare them pixel by pixel.
  • Several releases. The C64 version runs on an earlier engine (now called SCUMM version 0) with different scripts, different numbering and a different interface. The two DOS versions share scripts but not graphics. Each release is a cross-check on the others, and the NES, Atari ST and Amiga versions that came later each brought differences of their own.

That sounds like testing heaven. Two oracles! Several releases! And it mostly is. But a second implementation is still an implementation, and a good part of this post is about what happens when your oracle has bugs, versions and blind spots of its own.

How This Was Built

A few rules shaped everything regarding how I built this.

  • Port only what Maniac Mansion reaches. ScummVM’s SCUMM code covers dozens of games across a decade of engine versions. Maniacal follows only the branches that SCUMM versions 0 to 2, and this one game, actually run.
  • Tables come from tools, not typing. Any table copied from ScummVM (game fingerprints, fonts, opcodes, palettes, sound tables) is generated by a script that reads ScummVM’s source. A hand-typed table is a hand-typed bug.
  • Every phase ends with a check against something real: a screenshot of a known room, a known sound, or a scripted run that reaches a known state.
  • Don’t touch the reference. The ScummVM I compare against isn’t this project’s to change, so nothing about this work was allowed to modify it, its settings or its saves.

That last one sounds a bit fussy. It isn’t. An oracle you’re allowed to “fix” stops being an oracle.

Purpose and Audience

As with all posts in this series, there are two kinds of tester this is for.

  • Testers who explore by playing. The program, the game, a walkthrough you promise yourself you won’t read, and ScummVM open in another window.
  • Testers who explore the code. Testers who can read C# and, ideally, a little C++; using Maniacal‘s source, ScummVM’s source, the game’s scripts and a commented reconstruction of the C64 engine.

Both tracks matter more evenly here than they did for Laser Vault. A player can tell you that Edna caught a kid she shouldn’t have. Telling whether the script, my interpreter or the timing was at fault usually takes someone reading code. But telling whether it was supposed to happen at all often takes someone who knows the game.

What Correct Means

Here’s what a tester can lean on, roughly in order of authority:

  1. The game’s own scripts. They’re the specification of what the game wants. If the script says “when the kid opens this door, close it again and stop,” then that’s what should happen, however odd it looks. The repository has a commented disassembly of every C64 script in docs/scripts, with tables naming the rooms, objects and variables.
  2. The original engine. For the C64, the repository also holds a commented reconstruction of the original 6502 engine, all five hundred and some routines of it, in docs/c64-engine. When ScummVM and the original engine disagree, this is what settles it.
  3. ScummVM. It runs the same data. Maniacal writes its saves in ScummVM’s own format, so I can save at a moment, load that moment in ScummVM, capture its window, and diff the two screens.
  4. The other releases. If the C64 and both DOS versions agree with each other, that’s evidence. If one of them disagrees, that’s a lead.
  5. The documentation of the game. The manual, the hint books, a walkthrough, and a list of the game’s dead ends (more on those later).

Here’s what the comparison looks like in practice. On the left is ScummVM 2.7.0, on the right is Maniacal, both after loading the same save in Green Tentacle’s room:

Here’s a good test challenge! Before reading further, are those comparisons exact. Would this be a passing test?

To answer that, in the image, 250 pixels differ out of 64,000. All of them are accounted for: the cursor, which is in a different place in each window, and the sentence line (“Walk to”), which ScummVM 2.7.0 leaves blank after a load.

What I just said here is doing a lot of work, and it’s worth slowing down on. Every comparison in this project had a short list of expected differences, and a difference only counted as a pass if it was on the list and explained. “Close enough” is not a verdict. “These 250 pixels are the cursor and the sentence line, and nothing else” is.

Now for the complications.

The oracle has versions. Maniacal follows ScummVM’s current code. The ScummVM I first compared against was version 2.7.0. In between, in July 2024, ScummVM rewrote how kids walk in versions 1 and 2, to match the original more closely. So in the DOS releases, a kid sometimes stopped one column to the side of where ScummVM 2.7.0 stopped him. That was an expected difference. But notice what kind of expected difference it was: one I accepted from reading ScummVM’s history, not one I proved by running both versions of ScummVM’s walk. It was written down as exactly that.

Later I moved the comparisons to ScummVM 2026.4.0, which has that rewrite, and the newer oracle found things the older one couldn’t. The best of them: a kid facing left was drawn one column out of place. ScummVM 2.7.0 drew him one column out of place too, in exactly the same way, so until then the two implementations had compared perfectly while agreeing on a mistake. Two oracles that agree are evidence. They aren’t proof, especially when one was built from the other.

The oracle has blind spots. When you win, the ending plays in a special room, room 0. ScummVM 2.7.0 couldn’t load a save made there. It just said “Room 0: data not found.” So the last comparison of every route was made just before the end, not at it. The one moment you most want to compare is the one moment the reference can’t show you.

The specification has typos. The commented scripts in docs/scripts came from a different copy of the C64 game than the one I tested with. They agree everywhere but one place: picking up the Meteor in its room, where the script that runs checks an object’s state and the listing checks a variable. A small thing, but a good reminder that the “specification” is also an artifact somebody made from a particular copy.

The original is sometimes right where the reference isn’t. One example: on the C64, each room’s picture comes with a mask that decides what the kids walk in front of and what they walk behind. ScummVM stops decoding that mask eight bytes short. The original engine, according to its reconstruction, decodes the whole thing, and its last piece is used. Maniacal follows the original. That’s a case where “match ScummVM” would have been the wrong rule.

So the hierarchy isn’t “trust ScummVM.” It’s “use ScummVM to find differences, then let the game’s own code decide who’s right.” That’s the same lesson as MAME in the Laser Vault post, and I suspect it’s the lesson of every reference implementation ever.

Risk Areas

When I ranked the risks, what came out on top surprised me a little. It wasn’t drawing. Drawing is easy to check, because you can see it. It was everything about time.

Rank Risk Why it is risky here Tracks
1 Timing The house runs on timers, and most of them stand still during cutscenes. When Edna leaves the kitchen depends on how many cutscenes you’ve watched, not on the clock Both
2 Script semantics Each instruction’s exact behaviour matters, down to which script slot a new sentence reuses. A wrong guess usually still runs, just differently Technical
3 Several releases The C64 numbers its scripts, variables and objects differently from DOS, and has its own interface. A fix for one release can quietly break another Both
4 Walking Kids walk through a graph of boxes, and the C64 walks differently from DOS. Walks that are interrupted are where it goes wrong Both
5 Saves Saves have to round-trip in ScummVM’s format, including saves made in the middle of a running script Technical
6 Sound The C64’s SID chip, the PC speaker and the PCjr all sound different, and some sounds are over in a fraction of a second Both
7 Unwinnable states The game has deliberate dead ends. A bug that makes a winnable game unwinnable can look exactly like one of them Human
8 Toolchain and packaging A build configuration can change behaviour or speed without any code changing Technical

That seventh one deserves a moment. In most software, if the user gets permanently stuck, that’s a bug. In Maniac Mansion, it’s sometimes the design. Use the paint remover on the wrong thing, and you’ll never find the hidden door. Open the sealed envelope the obvious way, and you’ll rip it, and two of the endings are gone. A tester has to know which dead ends the game intends, because a bug that strands you looks the same from the inside.

Test Layers and Testability

The single most useful property of this project is that it’s deterministic. The engine never looks at the wall clock. It’s driven one tick at a time by whatever is hosting it, and the game’s random numbers come from a generator with a fixed seed. Give it the same game files and the same clicks on the same ticks, and it plays exactly the same game, every time, on every machine.

This is the same property that made Laser Vault testable, and it’s the same one I had to retrofit into Gauntlet and Improbable Mission. If there’s a single design choice I would tell anyone building something like this to make on day one, it’s this one.

That determinism gives these seams:

Seam What it makes possible
A headless host The whole game runs with no window, no sound device and no keyboard: in a test, or from the command line with –run
Scripted input Clicks and keys on given ticks, from a test or from the command line
Screenshots and recordings Any tick’s screen as a PNG and the sound as a WAV file, from the same run
Saves in ScummVM’s format Any moment of a test becomes a file ScummVM can load, so a test’s checkpoint is also a comparison point
Game state A test can read rooms, actors, objects, owners, variables and what’s been said, and fail with a description of all of it

And these are the layers built on them, 433 tests in all as I write this, which run in about a minute and a half:

Layer What it checks Track
Unit tests Decoders, tables, the save format, the walkbox search Technical
Real-data tests Every script in every release decodes; rooms, costumes and fonts render to pinned results, each pinned only after checking it by eye Technical
Interface tests The title, the kid selection and the intro, played by clicking Technical
Save tests Saves that round-trip, and saves that ScummVM itself wrote, loaded and compared field by field Technical
Walkthrough tests The whole game, won, by every kid who can win, in every release (the AI part, below) Both
ScummVM comparisons Saves from the walkthroughs loaded in ScummVM (2.7.0 at first, 2026.4.0 later) and the screens diffed Both
Package checks The published packages restored from a local feed, as a stranger would, playing each release Technical

There’s one thing about that table a tester should be suspicious of. The tests that need the game data return early when it isn’t there, because the game data can’t go in a public repository. That’s why the continuous integration build on GitHub can pass. But it also means that build passes with almost all of the interesting tests doing nothing at all! A green build there tells you the code compiles and the logic tests pass. It tells you nothing about whether the game actually plays.

This is the most dangerous kind of green: the kind that’s green because the test couldn’t look. If your test suite has tests that skip when a resource is missing, make sure somebody knows how many of them actually ran.

Heuristics for Exploratory Play

These need no code. My advice to a tester would be pick a few per session, and keep ScummVM open on the same moment if you can.

  • Let the house run. Stand still for ten minutes in different places and see who comes by. Edna, Ed, the doorbell, the mail and Dr. Fred all run on timers, and timers are where the risk is.
  • Count the cutscenes. Most timers stand still during cutscenes. Skip some cutscenes in one game and watch them all in another, and see whether the house’s schedule shifts the way it should.
  • Get caught on purpose. Being caught sends a kid to the dungeon with a cutscene, and the house carries on without him. It’s also, as you’ll see, sometimes the only way through a door.
  • Be somewhere when a cutscene starts. On a ladder, on the stairs, halfway through a door, in the middle of a sentence. Then try to carry on.
  • Try every kid’s skill. Bernard fixing the radio, Syd and Razor at the piano, Wendy at the typewriter, Michael in the darkroom. Then try each one with the wrong kid.
  • Walk into the dead ends. The repository’s docs/game/dead-ends.txt lists the game’s known ways to make itself unwinnable. Each one should be exactly as unwinnable as it was in 1987, no more and no less.
  • Save everywhere. Save in cutscenes, in close-ups, in the middle of a conversation, and load each one in ScummVM. Then load ScummVM’s saves in Maniacal.
  • Play the same moment in several releases. Same room, same action, different machines. Differences are leads, even when they turn out to be the releases genuinely differing.

Heuristics for Reading the Code

These are for testers who can read Maniacal‘s C# next to ScummVM’s C++ and the game’s scripts.

  • Follow only the branches the game reaches. ScummVM’s code is full of conditions on the engine version. For this game, only version <= 2 and the Maniac Mansion game ID matter. A port that took the wrong branch will often still work, just not like the original.
  • Read the jumps backwards. In the script listings I worked from, a conditional instruction names a condition and a place to jump, and the jump is taken when the condition is false. Read it the natural way and you’ll get every branch of a script exactly wrong. I know, because it caught us out more than once.
  • Translate the numbers between releases. The C64’s script, variable and object numbers aren’t the DOS ones. Some are offset by five or six; others follow no pattern. A test that hard-codes a DOS variable number will read the wrong thing on the C64, and it may not fail. The walkthrough reads the safe’s combination from variables 55 to 58 on DOS and 36 to 39 on the C64.
  • Decode the delays. Scripts store waits as negative numbers in 24 bits. A delay of 16759215 is really 18001 ticks, or five minutes. Knowing that turns a mysterious number into “the Meteor Police arrive five minutes after your call.”
  • Watch what a new sentence replaces. When a script starts an action straight away, the engine may run it in an existing script’s slot, including the script that asked for it. That’s a real behaviour, and it matters (see below).
  • When ScummVM and the original engine disagree, read the original. The C64 engine’s reconstruction is the tiebreaker, as with the mask that ScummVM decodes eight bytes short.

Sample Charters

As in the other posts, each charter is a target, some resources and something to discover. The time boxes are suggestions.

Charter Track Time box
Explore the kitchen at different times, with different numbers of cutscenes watched, to discover whether Edna leaves when the original says she should Human 45 min
Explore cutscenes that start while a kid is walking, on stairs, ladders and doorways, to discover kids that can’t move afterwards Both 60 min
Explore saving in unusual moments and loading in ScummVM, to discover state the save format loses Both 60 min
Explore each kid’s special action with every kid, to discover skills that work for the wrong kid or fail for the right one Human 45 min
Explore the dead ends list, to discover any that are more or less unwinnable than in the original Human 60 min
Explore the same scene in the C64 and DOS releases side by side, to discover where they differ and whether they should Both 45 min
Explore the sound of each release against ScummVM’s, to discover sounds that start late, end early or never play Both 30 min
Explore the script instructions that start sentences or other scripts, against ScummVM’s code, to discover slots reused differently Technical 60 min

Get an AI to Act Like a Maniac

In every post in this series, something (besides me, that is) has played the game. So the question for Maniacal was obvious: can it win? Not “does the first room draw,” but: can the whole game, start to finish, be won on this interpreter, by every kid who can win it, in every release?

And, as with Laser Vault, here’s the twist: the thing that plays is AI, but it’s AI of a particular sort. It doesn’t learn, and it doesn’t plan. It reads a walkthrough. Sort of.

A Player Who Reads

Think about how a person plays this game with a walkthrough open. The walkthrough says something like “Pull the door mat, pick up the key, unlock the front door with the key.” The person finds the verb “Pull” on screen, finds the door mat in the room, and clicks. If the door mat is off the edge of the screen, they walk the kid that way until it shows up.

And that’s what the test player, the AI agent, does. It plays by the names on the screen, not by object numbers or coordinates. A section of the route reads almost exactly like the walkthrough:

Under that, the agent player does what a person would. It clicks the verb, then the object where it actually appears on the screen, in the room or in the inventory. It waits for the kid to get there. It waits out cutscenes. When a room has two doors, “door[2]” means the second door from the left. And until part you see there is the agent player checking its own work: it repeats the action until the thing it was for has actually happened, because a cutscene can break in and take a click, just as it would take a click from a human player.

That’s intent, not implementation, which is the same thing I gave the Adventure agent. The route says what a player means to do. How to do it on this screen, at this moment, is the player’s job. That applies as much to the AI player as to the human one.

The route itself comes from a walkthrough in the repository, and it’s long. Syd’s way to win goes through the whole house: the silver key, the pool, Green Tentacle, the old rusty key, the man-eating plant, Ed’s package, Edna’s safe, the telescope, the microwave, the glowing key under the drained pool, the tools, the wires, Dr. Fred’s high score on Meteor Mess, a demo tape, a record contract, and finally the lab, the Meteor and a car that flies away with it. Along the way, the house’s timers have to be waited for or waited out: Edna leaving the kitchen, Ed’s lunch, the doorbell, the mail.

Dr. Fred’s high score on Meteor Mess is the combination to his lab’s inner door. He only plays once the arcade’s wires are fixed, and only when nobody’s watching. That’s a complicated bit of logic right there and the route has to wait for him.

If you want to see how the agent did, you can check out the video I put on my YouTube channel of the Maniac Mansion playthrough. Things you may notice when watching. The cursor jumps straight to each click instead of moving there, because the walkthrough clicks without moving the mouse. That is how a player could play it but most human players opt for the mouse. The AI player clearly optimized its actions. Also, this video is long: 68 minutes is the real route. But why that long? Well, it includes waits for timed events (Edna, the doorbell, the mail), so some stretches are just that: waiting.

Test the Instrument First

If you read the Adventure post, you know where this is going. The agent player is an instrument, and an instrument can be wrong. Several of the failures along the way were the agent player’s fault, not the interpreter’s:

  • Walking the kid towards an off-screen object, to click it as the second half of a sentence, quietly replaced the first half. The player now brings the second object on screen before it starts the sentence.
  • A double click meant to carry out “Use quarter with Meteor Mess game” built “Use quarter with quarter” instead.
  • On the 1987 DOS release, the doorbell is a short burst on the PC speaker, over before the player’s check for “is this sound playing?” ever saw it. The player now notes every sound as it starts instead of asking whether it’s still going.
  • Bernard gives a police badge to Purple Tentacle, and the player waited for the badge to leave Bernard’s hands. It never does. The game’s script reacts to the badge but lets him keep it. The route now waits for what Purple Tentacle says instead.

Each of those, on first sight, looked like an interpreter bug. None was. When the instrument and the system disagree, the instrument is the first suspect, especially when you built the instrument just recently!

And every time the AI player fails, it says what it saw. Here’s a real one, from when the badge route was first tried:


no "Purple Tentacle" for the second object, in room 31, camera at 8;
objects door#166, door#168, control panel#355, periscope#356, map of house#423;
actors Bernard at 34,57; carried package, hamster, card key, badge;
heard You're with the Meteor Police!! ... / AGGGGGGGG!!!!

Read that like a bug report. Purple Tentacle isn’t in the room. But we heard him say “You’re with the Meteor Police!!” and then scream. So the badge worked, he fled, and the player was still trying to give him a badge that he had already reacted to. A failure message that includes what the player heard made that a thirty-second diagnosis instead of an afternoon.

Every Kid, Every Release

Once Syd could win, the other endings followed:

Kid The way to win
Syd (and Razor) Record a song, trade it to Green Tentacle for his demo tape, mail it to a record company, and give him the contract. He’ll protect you from Purple Tentacle. Send the Meteor away in the car
Bernard Fix Dr. Fred’s radio and call the Meteor Police. They take the Meteor, and drop a badge that scares Purple Tentacle off
Wendy Rewrite the Meteor’s terrible memoirs and mail them to a publisher. The contract gets you past Purple Tentacle and talks the Meteor out of crime
Michael Give Ed his package, develop the film of Ed’s plans that turns up outside, and give him the prints. Ed takes on Purple Tentacle himself

Dave and Jeff can’t win on their own, and Razor’s way is Syd’s. So that’s four ways to win, in three releases: twelve complete games, each one played from the driveway to “You won the game!” A whole game, an hour or so of the house’s time, plays headless in well under a minute. Later the Atari ST and Amiga versions were won all four ways too, and the NES Syd’s way, which makes twenty-one.

Think about the value of this from a regression testing perspective! Would a human tester really replay the games with all of the steps, across all versions? Of course they wouldn’t.

The endings share most of their route. All of them need the lab’s doors open, and getting there is most of the game. So the tests share that part of the route, and then branch. That’s also what made the other teams possible: the route names kids by their part, not by name. “The kid picked first” does what Syd does in Syd’s game. In Wendy’s game, Wendy does it.

And every section of every route saves its state as a ScummVM save file, 28 of them per release in those first three. Each one was loaded in ScummVM and its screen compared with Maniacal‘s. In every case, the differences were the expected ones.

Here’s the same room from the first three releases, the C64, DOS 1987 and DOS 1988, from those saves:

Case Study: The Kid Who Couldn’t Climb Down

This is my favorite bug of the project, mainly because it wasn’t one.

To get a glowing key, you drain the swimming pool, climb down the ladder into it, grab the key and get out before the nuclear reactor it cools overheats. You can’t take long. Draining the pool sets off an alarm, with a cutscene of Dr. Fred discovering his reactor overheating, and two minutes later the house melts down.

On the C64, Syd climbed down the ladder, the alarm’s cutscene broke in, and when it was over, Syd was stuck. Not caught, not dead. He just stood on the ladder, and no click could make him move again. Up, down, anywhere. And the clock on the meltdown was still running.

The obvious verdict was a walking bug in Maniacal. The C64 has its own walking code, and the C64 was the last of those three releases Maniacal took on. So the first step was the oracle. I saved just before the alarm and loaded that save in ScummVM 2.7.0. Same result. Syd stood on the ladder in ScummVM too.

Then the code. When a cutscene interrupts a kid’s walk on the C64, the kid is left standing, but the next point of his walk is kept. And when a new walk starts, if its first point is the one the kid already had, the walk does nothing at all, because the engine thinks the kid is already on his way there. On most of the floor, that never matters, because there are lots of ways to go. But on a ladder, every way down begins at the same point: the foot of the ladder. So every new walk is refused. Syd is stuck for good.

Maniacal and ScummVM agree on this, line for line, and ScummVM’s C64 walking follows the original engine. So then what I did is just run the original game on a Commodore 64. I saw the same thing and that proved it’s not a Maniacal bug. It’s a bug from 1987.

Which means it isn’t mine to fix. Recall the rule from the Improbable Mission post: if the original behaves the same way, it’s not a bug in my version. Record it as a quirk of the original.

So the route changed instead of the engine. Syd waits at the side of the pool until the alarm’s cutscene is over (skipped with Escape, as a player may), and until Purple Tentacle has been sent down to check the cellar. Then he climbs down.

And then, much later, it happened again. In Michael’s route, on the C64, Michael was walking up the stairs from Green Tentacle’s landing when Dr. Fred’s second visit to Sandy in the lab came on. That’s a cutscene that plays forty-five minutes of the house’s time after his first. When it was over, Michael was stuck on the stairs. Same quirk, different stairs. This time the diagnosis took minutes, not hours, because it had been written down. Michael now waits for Dr. Fred in the darkroom.

Before moving on, let me show you something interesting here. As I was writing this post, I realized I never captured a screenshot of the situation I just described. This might be just like a tester who did a complex test and didn’t capture the results. And now really hates the idea of running the entire test again. Well, here’s where I turned to my agent. I told the agent what I wanted (my intent was to show the bug in action) and this is what it came up with:

The log.txt it generated was this:


alarm 121 steps after Syd is chosen again
lead 5: at 9,64 as it broke in, 18,37 after, 18,8 in 2 after a click, stuck False
lead 10: at 14,59 as it broke in, 14,59 after, 14,59 in 6 after a click, stuck True
lead 15: at 15,56 as it broke in, 15,56 after, 15,56 in 6 after a click, stuck True
lead 20: at 15,51 as it broke in, 15,51 after, 15,51 in 6 after a click, stuck True
lead 25: at 16,46 as it broke in, 16,46 after, 16,46 in 6 after a click, stuck True
lead 30: at 16,41 as it broke in, 16,41 after, 16,41 in 6 after a click, stuck True
lead 35: at 16,37 as it broke in, 16,37 after, 16,37 in 6 after a click, stuck True
lead 40: at 21,37 as it broke in, 21,37 after, 21,37 in 6 after a click, stuck True
lead 45: at 26,37 as it broke in, 26,37 after, 26,37 in 6 after a click, stuck True
lead 50: at 31,37 as it broke in, 31,37 after, 31,37 in 6 after a click, stuck True
lead 55: at 33,37 as it broke in, 33,37 after, 33,37 in 6 after a click, stuck True
lead 60: at 36,37 as it broke in, 36,37 after, 36,37 in 6 after a click, stuck True
lead 65: at 36,38 as it broke in, 36,37 after, 36,37 in 6 after a click, stuck True
lead 70: at 36,41 as it broke in, 36,41 after, 36,41 in 6 after a click, stuck True
lead 75: at 36,43 as it broke in, 36,43 after, 36,43 in 6 after a click, stuck True
before the gate: playable True, cursor True, room 6
gate clicked: playable True, kid 36,41, sentence verb 0
  +10: kid 30,37 in 6, screen room 6, cursor True, moving 4
  +20: kid 20,37 in 6, screen room 6, cursor True, moving 4
  +30: kid 16,40 in 6, screen room 6, cursor True, moving 2
  +40: kid 16,40 in 6, screen room 6, cursor True, moving 2
  +50: kid 16,40 in 6, screen room 6, cursor True, moving 2
  +60: kid 16,40 in 6, screen room 6, cursor True, moving 2
  +70: kid 16,40 in 6, screen room 6, cursor True, moving 2
  +80: kid 16,40 in 6, screen room 6, cursor True, moving 2
  +90: kid 16,40 in 6, screen room 6, cursor True, moving 2
  +100: kid 16,40 in 6, screen room 8, cursor False, moving 2
  +110: kid 16,40 in 6, screen room 8, cursor False, moving 2
  +120: kid 16,40 in 6, screen room 8, cursor False, moving 2
  +130: kid 16,40 in 6, screen room 8, cursor False, moving 2
  +140: kid 16,40 in 6, screen room 8, cursor False, moving 2
  +150: kid 16,40 in 6, screen room 8, cursor False, moving 2

From this I could extract a few images.

This is the last frame before the alarm’s cutscene takes over, with Syd partway down.

This is just after the cutscene and Syd is in the same spot, with the player having control again.

This is the issue: Syd is on the ladder in the drained pool, “Walk to ladder” is on the sentence line with the cursor on the ladder, and he doesn’t move.

Not only was the AI able to reproduce this on demand, not only could I grab screenshots from that, but it also created a file called maniac64.s20, which was a save of the stuck state. This means I could simply restore to that position in the future without having to walkthrough everything. For ScummVM, I could copy that save into the target’s save folder and load slotĀ 20.

Bugs That Weren’t Mine

The ladder wasn’t the only time the game itself was the culprit. A few more, briefly, because each one teaches the same lesson from a different side:

  • The door that closes on itself. On DOS, walking from the cellar through the door to the dungeon doesn’t work. The door’s script closes the door behind the kid by starting a sentence at once, and that sentence takes over the script’s own slot, so the rest of the script, the part that takes the kid through, never runs. ScummVM leaves the kid in the cellar the same way. The route gets into the dungeon by being caught instead, which, as a player, is how you’d do it anyway.
  • The dimes that wouldn’t come. Breaking Ed’s piggy bank gives three dimes, but picking one up straight away did nothing. The script that hands out dimes is the one still showing “Damn! I broke it!”, and starting it again while it waits on that line does nothing. ScummVM agrees. The player now waits for the line to finish.
  • The oven that stopped. Steaming the envelope open in the microwave only works if you leave the oven alone until it’s done. Open it early and its script stops, and so does the cooking. That’s the game, not the engine.

Every one of those would have been easy to “fix” in Maniacal. And every fix would have made Maniacal degrade a quality: namely faithfulness. A fix would mean the game was less faithful to the original. When you’re building something to match an original, the urge to fix the original’s behavior is the urge you most need to resist.

And One Bug That Was Mine

To be fair to myself, plenty of bugs were mine. Here’s one I like, because no code changed. Late in the project, Maniacal stopped using the shared host libraries it had started out with and took its own copy of the parts it used. The code was identical. The tests all passed. And the test suite went from two and a half minutes to seven.

The libraries had come as optimized release builds. The copied code was now compiled as part of a debug build, and the hottest code in the whole project, the code that touches every pixel, was suddenly running without any optimization at all. The fix was one line in a project file, with a comment saying why. Afterwards the suite took under a minute, faster than before.

That’s toolchain drift again, the same thing that bit me in the Adventure post. Nothing about the code changed. Everything about how it was built did.

Heuristics for Judging Agent Results

The walkthrough player isn’t perhaps AI in any interesting sense, but a few points came out of using it that apply to anything you point at software where the agent plays it for you:

  • A win proves a route, not the game. The agent player proves these routes work, in this order, with these timings. A different order might hit something the routes never touch.
  • Check the reference before blaming the engine. When the agent player gets stuck, the first question is whether the original gets stuck in the same place. Here, it often did.
  • Suspect the instrument when it’s new. Most of the early failures were the agent player’s misunderstanding of the game, not the interpreter’s.
  • Make failures talk. A failure that says where it was, what it carried and what it last heard is worth ten that just say “timed out.”
  • Know what your green means. A suite that skips without its data, or a comparison that excludes “expected” differences, is only as good as the list of what it skipped.
  • Write the quirks down. A quirk of the original, found once and recorded, turns the next encounter into a five-minute job.

Game Over

When I started this, I wanted to know whether a reimplementation of a 1987 engine could play that game well enough for the game not to notice. Twenty-one complete games later, with every section of every route checked against a second implementation, I’m fairly confident it can.

But the thing I’ll remember is how often the answer to “is this a bug?” was “yes … in 1987.” Testing against an original means accepting the original, ladders and all. And having a second implementation to compare against turned out to be less like having an answer key and more like having a colleague: someone whose opinion matters a great deal, who is usually right, but who still needs checking.

Plus, Sandy finally got out of that lab. Four different ways, on five different machines, which is five machines more than Dr. Fred planned for.

What’s Next?

Well, we’re not done with the mansion quite yet. Believe it or not, I built an API into this project and that will let me talk about API testing in this context. Stay tuned!

Share

This article was written by Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.