It’s definitely time to talk seriously about testing artificial intelligence, particularly in contexts where people might be working in organizations that want to have an AI-enabled product. We need more balanced statements of how to do this rather than the alarmist statements I’m seeing more of. So let’s dig in!
Author: Jeff Nyman
Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words.
If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.
Creating Explanations: The Ethos of Testing
A couple of years ago I talked about what I considered to be the basis of testing. I very confidently asserted things. Maybe I sounded like an authority. Maybe I sounded like I was presenting a consensus. But did I really talk about the basis or just a basis? And shouldn’t an ethos have been part of that basis?
Continue reading Creating Explanations: The Ethos of Testing
The Economics, Value and Service of Testing
Among the many debates testers have, one of those is whether it makes sense to write tests down. Sometimes this is framed, simplistically, as just writing down “test cases” and, even more simplistically, as a bit of orthodoxy around how you don’t write tests, you perform tests. So let’s dig into this idea a little bit because I think this seemingly simple starting point for discussion leads into some interesting ideas about what the title of this post indicates.
Continue reading The Economics, Value and Service of Testing
Exploring, Bug Chaining and Causal Models
Here I’ll go back to a game I talked about previously and show some interesting game bugs, all of which came out of exploration and where the finding of one bug guided exploration to finding others, which led to some causal mapping. Of course, the idea of “bug chaining” and “causal mapping” is certainly valid in any context, not just games. But games can certainly make it a bit more fun!
The Emic and Etic in Testing
There’s an interesting cultural effect happening within the broader testing community. I’ve written about this before, where my thesis, if such it can be called, has been that a broad swath of testers are using ill-formed arguments and counter-productive narratives in an attempt to shift the thinking of an industry that they perceive devalues testers above all else. This has led to a needlessly combative approach to many discussions. In this post I want to approach this through a couple of parallel lenses: that of game studies, linguistics, and anthropology. That will lead us to insider (emic) and outsider (etic) polarities. It’s those polarities that I believe many testers are not adequately shifting between.
A History of Automated Testing
What I want to show in this post is a history where “teaching” and “tutoring” became linked with “testing” which became linked with “programmed instruction” which became linked with “automatic teaching” and thus “automatic testing.” The goal is to show the origins of the idea of “automating testing” in a broad context. Fair warning: this is going to be a bit of a deep dive.
The Breadth of the Game Testing Specialty
I’ve posted quite a bit on game testing here, from being exploratory with Star Wars: The Old Republic, to bumping the lamp with Grand Theft Auto V, to ludonarrative in Elden Ring. I’ve also shown how comparative performance testing is done with games like Horizon Zero Dawn. These articles offered a bit of depth. What I want to do here is show the breadth of game testing and some of the dynamics involved since it’s quite a specialized sub-discipline within testing.
Testing: From Aristotelian to Galilean, Part 2
In this post I’ll continue on directly from part 1 where we ended up with a lot of models and a recognition of competing interpretations of quality along with a need for testability.
Continue reading Testing: From Aristotelian to Galilean, Part 2
Testing: From Aristotelian to Galilean, Part 1
Any discipline can focus along a spectrum of thinking. That’s no less true of testing, of course. The spectrum I want to introduce from history is that of moving from an Aristotelian to a Galilean way of thinking and “doing science” which, in many ways, is synonymous with “doing testing.”
Continue reading Testing: From Aristotelian to Galilean, Part 1
When Testing Questioned Orthodoxy
When Testing Questioned Philosophy
In the first post in this series, I ended by focusing a bit on Galileo who started to make the idea of testing what it eventually would be recognized as today. That’s the same thing as saying Galileo effectively produced one of the first attempts to make science as it is known today. Let’s continue this path of investigation.
When Testing Became Scientific
As I’ve been teaching the history of science and religion recently, some interesting ideas have formed in my head around how to present certain topics as they relate to testing. This is crucial since testing is the basis of effective experimentation. So here I’ll talk very briefly about how testing truly became testing.
Testers, We Need a Narrative
I was recently re-reading Houston: We Have a Narrative by Randy Olson and I was struck by certain concepts there that reminded me how poorly framed testing often is, particularly by its own practitioners. Clearly an opinionated statement, of course, but I very much believe that many testers in the industry currently lack a narrative or are using a malformed narrative. And this is hurting the industry more broadly as we see quality problems get worse and worse.
Ludonarrative Testing, Part 3
In the second post of this series I looked at a couple of games to drill in the idea of ludonarrative and what it means. Here I want to go back to a game I started with in the first post, Elden Ring, and take a much deeper look at the mechanics and the narratives from a ludonarrative testing standpoint.
Ludonarrative Testing, Part 2
Continuing on from the first part, I want to continue to give testers a look into a very specific, and often undocumented, form of testing in the context of games, which is the idea of ludonarrative. This has the benefit of also showing how quality can be very much a function of viewpoint.
Ludonarrative Testing, Part 1
I’ve long said that I do believe game testing is one of the best ways that testers can improve their skills. Yet there’s very little out there that’s substantive about game testing, particularly in terms of how testers are asked to think beyond just “test that the game works.” So let’s dig in to this a bit with a two part series that involves something even a lot of game testers seem unaware of, which is the concept of ludonarrative.
Don’t Be A Model Literalist
A recent discussion came up around a particular type of product model and I wanted to cover that a bit here since I’ve find a certain type of thinker — the literalist — will tend to have problems using imagination and abstraction with these product models. I’ve also found this translates to other models, such as those about testing.
The Testing Pedigree
Testers tend to debate a lot as to whether “everyone” can be a tester. The answer is: yes, if you are human, you are automatically a tester. But not everyone is someone who specializes in testing. I talked a bit about this in being a test specialist and whether or not we should hire test specialists. Here I just want to back up that notion that if you are human you test by looking at the pedigree of the concept.
Testing at the Crime Scene, Part 4
In the previous post we did some deep dives, using all the techniques of this series so far, to try and get a feel for the overall landscape of a code repo to look for clues. Now let’s start to narrow our focus again a bit and then wrap up this series with a few points about the journey we’ve taken.
Testing at the Crime Scene, Part 3
In this third post to the crime scene series, we’re going to continue using our crime scene techniques by adding an extra complexity dimension to what we started in the second post. We’re then going to try our analysis on a much larger code base than any we’ve looked at so far. So put on your detective hat and let’s dive in!