Laser Vault: Under the Hood

In the last post, I talked about testing Laser Vault, my emulator for the laserdisc arcade games of the 1980s. That post was about strategy: what “correct” means when the game is the original program, where the risks are, and how autoplay ended up as the best test I had. This post goes underneath all that. It’s about how the thing actually works, and about the instruments I built to look inside it.

Continue reading Laser Vault: Under the Hood →

Testing with Lasers

So far in this series, I’ve tested Adventure Uncartridged, which had an available original to compare directly against, Gauntlet Reforged, which had no available original at all, and Improbable Mission, which had an original but, this time, the original isn’t something I compare against. It’s the actual program I’m running.

Continue reading Testing with Lasers →

When Performing Causality Means Performing Experience

I originally wanted to title this post “What Is It Like to Be Nothing?” which would be a direct riff on Thomas Nagel’s famous paper “What Is It Like to Be a Bat?” Any philosophically literate reader will recognize that immediately but, in the end, I wanted to continue the theme from my previous post and talk about wider implications.

Continue reading When Performing Causality Means Performing Experience →

When AI Performs Causality Instead of Practicing It

In my previous post, I talked about the cautionary aspect of AI hallucinating its own hallucinations. There’s a deeper element to this and it has to do with causality. Or, rather, the lack of it. In this post, I’ll look at how to test this aspect of AI in very simple way, which will point to a deeper way to consider how to consider the trustability of AI when cause and effect matter.

Continue reading When AI Performs Causality Instead of Practicing It →

When AI Hallucinates Its Own Hallucinations

I recently had an ongoing series on AI and Testing. Today I’m not running evals or measuring correctness scores. Today I’m doing something more uncomfortable: watching an AI fall apart in real time, and narrating what’s actually happening at each step.

Continue reading When AI Hallucinates Its Own Hallucinations →

AI Was Already Here: Loud Opinions vs. Precise Understanding

Previously I had talked about the idea of personal marketability when it came to learning AI. That was in the context of an AI and Testing series, so I trust what was being learned was obvious. I’m seeing an interesting trend developing with ardent opponents of any and all AI, particularly on social media platforms like LinkedIn. This is my attempt at a social response to that.

Continue reading AI Was Already Here: Loud Opinions vs. Precise Understanding →

DSPy: Declaring Instead of Prompting

In my AI and Testing series, which ran for a couple of months, I focused heavily on the testing side of things. I now want to consider AI in some specific contexts that you are likely to come across and show how those contexts work. The first that I’ll focus on is a tool called DSPy.

Continue reading DSPy: Declaring Instead of Prompting →

AI and Testing: Evaluation Synthesis

This series has now covered eight metrics across two evaluation paradigms. We’ve applied them to a warp drive paper, a mass extinction paper, a philosophical essay on colliders and cosmology, and an essay on time travel and metaphysics. Each post introduced a metric, showed what it catches, and explained what its scores mean in context. This post does something different: it stops introducing new metrics and instead asks what the full set reveals when used together.

Continue reading AI and Testing: Evaluation Synthesis →

AI and Testing: Evaluating Conversations

In the previous posts in the DeepEval series, we built up a diagnostic framework for evaluating RAG systems, covering Faithfulness, Contextual Precision, Contextual Recall, Contextual Relevancy, and G-Eval. All of those metrics operate on single turns: one question, one retrieval context, one response, one score. In this post we’ll move into different territory: conversational evaluation.

Continue reading AI and Testing: Evaluating Conversations →

AI and Testing: Recall, Relevancy, and Richer Evaluation

In the previous posts we looked at the Faithfulness and Contextual Precision metrics with DeepEval, and started building an intuition for how retrieval failures cascade into generation failures. Those two metrics told us what went wrong and where in the pipeline. In this post, we’ll add three more tools to the diagnostic kit: Contextual Recall, Contextual Relevancy, and G-Eval.

Continue reading AI and Testing: Recall, Relevancy, and Richer Evaluation →

The Last Useful Animal

In my previous posts, I’ve been talking a lot about AI technology and tooling, and any enthusiasm within those posts came from helping people see how and where to test AI, and keep it explainable, interpretable, testable and, thus, ultimately trustable. All that content being said, I have serious concerns about what AI is potentially doing to us: not to our test pipelines, but to us as a civilization. Fair warning: this is a bit of a Thomist indictment of techno-oligarchy.

Continue reading The Last Useful Animal →

Testing the “Yes-Man” in Your Pocket

If you’ve been following my recent posts on how to test AI, you know that evaluating Large Language Models (LLMs) requires an entirely different mindset than traditional software testing. We’re no longer just testing for crashes, latency, or even factual hallucinations. As AI becomes deeply integrated into our daily lives, we have to start testing for psychological and behavioral impacts.

Continue reading Testing the “Yes-Man” in Your Pocket →

AI and Testing: From Specification to Story

In the previous post, we built a formal ontology from the Z-Machine specification, used it to drive code generation, and then turned it into a test oracle. At the end of that work, I hinted that the next step might be to put an LLM directly in the loop and watch it actually play one of those games. That’s what this post does.

Continue reading AI and Testing: From Specification to Story →

AI and Testing: From Ontology to Implementation

In the previous post, we looked at setting up an ontology based on a Z-Machine specification. Our goal was to get this in place so that we could have an LLM generate code to implement the portion of the ontology that we described. In this post, we’ll attempt exactly that.

Continue reading AI and Testing: From Ontology to Implementation →