Exploration – Testing and Checking

In this post I want to follow on a bit from the interactive exploration idea developed up to this point but also focus on the distinction of checking and testing that often gets debated. I also want to use this post to reinforce a few things I talked about last year.

If you check back to the previous post, you’ll see the section titled “Exploration Frames Testing — and Checking”. What I want to do here is show you a solution to the business scenario that was presented and then use that to expand on that framing idea.

Hopefully by this point you found yourself with enough free time to check out and play around with the game scenario on your own plus took some time to think about what you would test given the business scenario. Let’s first consider a logical thing some folks might be thinking.

Was this all unfair to readers?

I want to state that I developed the code for this game as I was writing these posts. That’s partly why it’s taking so long between posts. And my reason for bringing that up is that I genuinely had no special knowledge beyond where I wanted the posts to go. I was exploring right along with anyone reading these posts. And, as you’ll see, I encountered a bug in Inform during my explorations and I also found that one of the elements I presented in the previous game (release 3) was very inadequate. I also found that I used a property that I probably didn’t need to.

This was testing and checking

(Note: as some of my readers know, I’m not a fan of the testing vs checking distinction. But here I went with it as a way of seeing if I could be convinced.)

All of this was me exploring as a developer (of the game) and as a tester (of the game, but also of the blog content). Further, this led to me doing a series of checks that backed up my explorations. These were checks I had to, in some cases, refine based on feedback from further exploration. Given that I’m not presenting this to readers live and in-person, the best I can hope for is to showcase the results of the work.

But please do notice, by the way, that these are historical results. History matters! This work on the game has gone through a project singularity and you are left to reason about what has been left behind. You’ve also seen a bit about using the code as specification.

Let’s Consider the Game

For this, we’re looking at release 4 of the game.

This was partly a challenge about considering the buying action and you can see the implementation of that in Part – Buying.

You might want to consider that in contrast to and comparison with the photographing action we did in Part – Photographing in release 2 of the game.

There’s actually an interesting little non-obvious problem with the following:

Understand "buy [something]" as a mistake ("That's a bit vague. Try 'buy [the noun] from [a random person that is not the player]' or, if it's clear who will be the subject of the action, try 'buy [the noun] with [a random thing that is currency]'.")

You would have to try it out to see what that might mean. Hint: you might want to consider how Inform models kinds, which you can remind yourself of by looking at the section “Let’s Talk Modeling” in the first post in this series.

The Core Testing Focus

The main thing to look at here is the Section – Buying the Bag. I left a series of comments in the source text including some separators just for ease of reading. There are a few things that are interesting to me here.

First, notice that even though we have source code that is in English, it still helps to have descriptions around the code and the tests to explain exactly what is happening.

Important Point: If that’s the case with source code that is very close to natural language, how much more so do such representations falter with BDD-style approaches, such as with Gherkin and Cucumber?

Interleave Testing and Checking

Notice how at some points I interleaved the checks (the “test” statements) with the functionality. I did that primarily to give you a visual — albeit after-the-fact — of how I performed exploration of the game and then, when it seemed relevant, encoded some of that exploration into scripts. I shifted between polarities of testing and checking; of going without a map and then of creating a map.

These were interleaved activities and while I do believe there is value in the operational distinction of testing and checking, I think it’s very possible for these to become semantic debates rather than just getting something done. On the other hand, you could argue it does pay to know when you are doing one versus the other. I believe there’s some truth to that, but I also believe that experienced testers do this automatically by being proficient at interleaving activities. It’s part of the discipline that they have internalized.

I’m hoping fun examples like the one I’m showing here — well, fun to me, anyway — are a way for experienced testers to help new testers build up their intuition about how to do these activities, particularly in an interleaved manner. We need to have an intuition for abstraction while also realizing there are limits to our intuition when it comes to certain techniques.

Buried in this section of code, you’ll see this:

Instead of taking the bag of bird crumbs:
	[if the aged woman is compensated:]
Instead of taking the small coin:
	[if the aged woman is compensated:]

It turns out that having the notion of “compensated” isn’t really necessary. This was not something I planned for when I wrote these posts or the game example. As I was exploring how to construct the game, and the tests for this post, I realized I did a bit of over-engineering. And I realized this by interacting with source code as a specification as well as an understanding of the business scenario that I was operating with, which only required a certain level of simulation.

As you’ll see, I also discovered a bug in Inform 7 itself. (Or, rather, discovered a bug that was already known about.) Check out Book – Rules for that:

Handles a bug.
See: http://www.intfiction.org/forum/viewtopic.php?f=7&t=21118
And: http://inform7.com/mantis/view.php?id=1969
The exclude indirect possessions from take all rule is not listed in the for deciding whether all includes rulebook.
Rule for deciding whether all includes things enclosed by the person reaching while taking or taking off: it does not.

This was not a bug I was aware of and I provide links to the forum thread where I asked about it as well as the bug itself. Again, this was a case of me exploring the very thing I’m using to help testers explore!

Revising Ideas Based on Exploration

Originally when I wrote one part of the logic for this post, I had (in the Section – Context) a bit that I’ll reproduce here:

Every turn when the player is in Broad Walk:
    if a random chance of 65 in 100 succeeds:
        say "[one of]'Thirty p! Thirty p a bag!' cries the bird woman.[or]'Feed the birds! Thirty p!' Her voice quavers with heartbreak.[or]'Feed the hungry birds!' hollers the bird woman.[or]'Thirty p for the starving birds!' calls the bird woman.[or]'Feed the birds!' yells the bird woman.[or]'Thirty p!'[at random]"

But as I explored my own game, I realized this wasn’t good enough and you see the extra line I had to add in the source:

unless the aged woman is providing a paid for product:

The problem was I realized that by paying the woman — and when she was thus “providing a paid for product” — she would still be yelling out that she had food to sell, which seemed odd for her to be doing while interacting with me, her customer. Plus it made the output look a little messy.

Exploring Conditions

Consider the section Chapter – Getting the Bag. Originally, while exploring the idea, I developed the following (which you can still see as a commented out bit in the source text):

Every turn during Giving the Bag:
    unless the player has the bag of bird crumbs:
        if the player is in the location of the aged woman:
            say "[one of][run paragraph on][or]'Take yer bag and change, guv'ner!'[or]The bird woman tugs your sleeve impatiently. 'Take yer bag, guv'ner! Ain't got all day!'[or]'You don't take it, I be keeping it.'[or][run paragraph on][stopping]"

That, however, upon exploration, was found to be woefully inadequate. I ended up — via working with “developer” (me) and “tester” (also me) — coming up with what you see:

Every turn during Giving the Bag:
  if the player is in the location of the aged woman:
    if (the aged woman encloses the small coin) and (the aged woman encloses the bag of bird crumbs):
      say "[one of]'Take yer bag and change, guv'ner!'[or][run paragraph on][stopping]";
    if (the aged woman encloses the small coin) and (the player encloses the bag of bird crumbs):
      say "[one of]'You be wantin['] your change, then, right?'[or][run paragraph on][stopping]";
    if (the aged woman encloses the bag of bird crumbs) and (the player encloses the small coin):
     say "[one of]The woman tugs your sleeve impatiently. 'Take yer bag, guv'ner! Ain't got all day!'[or][run paragraph on][stopping]"

I’ll leave it for you to determine why this necessary. I think you’ll be able to figure it out.

Domain Concepts

By the way — did you notice, even from the past example, that the chapter heading is “Getting the Bag” but the scene was called “Giving the Bag”? A little discrepancy to note as you explored. You might have also noticed that one section is called “Bird Feed” but that’s never really mentioned in the code. It’s just “crumbs” and “bags of crumbs”. Come to think of it, the section for the aged woman is called “Bird Woman”.

And, yes, this stuff can matter. Have you ever been on a project where someone was talking about “Customers” and someone else was talking about “Accounts” but, it turns out, in the system they are both the same thing? Or the opposite: people were talking about “Customers”, “Accounts”, and “Users” as the same things but they were actually entirely different. And the thing people were calling “Accounts” were in fact called “Entities” in the system.

Being able to have a domain vocabulary and have that reflected in the sources of truth (of which you hopefully have a minimum of) is an important part of concepts like domain-driven design or the creation of ubiquitous languages.

How Things Appear

You might notice how the description of the aged woman had to become a bit more conditionalized (under Section – Bird Woman):

The description of the aged woman is "[if the aged woman is providing a paid for product][purchase-state].[otherwise]You get the feeling that she's been selling crumbs on this same bench, year after year, since well before you were born. Her face is lined with care for her feathered charges, who perch on her round shoulders without fear."

Which required this addition:

To say purchase-state:
  if (the aged woman encloses the small coin) and (the aged woman encloses the bag of bird crumbs):
    say "She's waiting for you to take the bag you purchased and your change" instead;
  if (the aged woman encloses the small coin) and (the player encloses the bag of bird crumbs):
    say "She's waiting for you to take your change" instead;
  if (the aged woman encloses the bag of bird crumbs) and (the player encloses the small coin):
    say "She's waiting for you to take your bag" instead.

Again, perhaps you can see why this was necessary for a more complete experience. Again, though, this came out of exploring the game.

Oh, by the way, did you notice something interesting about the “To say purchase-state” conditions and the conditions I gave about “Every turn during Giving the Bag”? Take a look at the conditions. What do you notice? That might be something a developer is interested in but, you as a tester, also should be interested.

Did you spot it?

The conditions are the same. But we have two basis paths for the same functionality, which brings up the ideas of path testing. There are two spots for this code to go wrong, which means I have to test both as if they were separate. Whereas if I know the same code is being called, deciding what tests to run for regression might be simpler. What the above is showing you is a sensitivity in the implementation and testers should be concerned about various sensitivities.

So even if the developers did not think to put the duplicated conditions in the equivalent of a method, the testing put pressure on the design such that it suggested it.

There are a few other little things that were added to the source code by the developer (“me”) based on some comments by the tester (“me”). You may want to explore the source text for yourself and check it out.

The End Goal

What I was ultimately exploring and testing around was the following:

  • The aged woman in the Broad Walk is selling bird crumbs.
  • The player needs to buy a bag from her.
  • She will only accept local currency and that must be thirty pence of greater.
  • If purchased, the woman will offer the bag and some change for a limited time.
  • If, in that time, the player doesn’t take the bag, they lose their chance to ever buy it.

A bare minimum way to at least check some of the interactions involved is provided in the solution. But this is just one acceptance-level scenario that does not go anywhere near far enough in terms of making sure the game responds well to the situation as it was implemented.

So we have here a series of lower-level tests (the “test” statements, which you can try out in the working game available via the site). Notice how you might call those “unit”, “integration” or even “integrated” depending on your view. The fact that they sit close to the code should not be what decides the layer. That’s a circumstantial aspect rather than a situational one. The solution that I proposed via the solution file could have been encoded as a test statement was well:

Test buying with "north / look at the woman / buy bag from woman / take the bag / look at the woman / take the change".

You might want to think about how to create more such statements, and perhaps try them out in the game which, again, is available to play via browser from the game site.

First Experiment Concluded!

And that’s it, folks! This was the first attempt to provide an interactive fiction exploration example to help people, including myself, think about exploration, testing, and checking.

For me, this was a lot of fun and it reinforced in me the power of this approach, even when not being done in-person. I do plan to explore more ideas around these interactive examples throughout this year, using what I learned here as a way to better refine how I present this material via blog posts. I hope you’ll join me along the way.

In the meantime, happy exploring!


This article was written by Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.

One thought on “Exploration – Testing and Checking”

  1. I discovered this series a bit late but loved it.  I come from an 80s BASIC programming backgrou nd with the old TRS-80 computer.  Inform is a programming gem, or should I say ‘red ruby’. I find the exact same concepts to hold true from programming in my youth.  Your series here is very thought provoking, albeit a little over my head at times, but it was extremely thorough and helped me wrap my head around approaching these software problems.  Bravo.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.