Author: Jeff Nyman

Anything I put here is an approximation of the truth. You're getting a particular view of myself ... and it's the view I'm choosing to present to you. If you've never met me before in person, please realize I'm not the same in person as I am in writing. That's because I can only put part of myself down into words. If you have met me before in person then I'd ask you to consider that the view you've formed that way and the view you come to by reading what I say here may, in fact, both be true. I'd advise that you not automatically discard either viewpoint when they conflict or accept either as truth when they agree.
Published inTesting

Testing Knowledge Isn’t (Just) a Testing Skill

written by

The first and second posts in this series were about one person occupying two roles on the same project, sometimes switching between them within a single bit of work. That’s interleaving, and it works because the switch is cheap: same person, same context, same code sitting right there to look at. But most of the testing world doesn’t look like that. Most of it looks like two different people, on two different sides of a task, who never actually swap seats. I want to look at what “the role, not the person” demands in that arrangement: not interleaving so much, but cooperation. And I want to do it with something small enough to show in full.

Continue reading Testing Knowledge Isn’t (Just) a Testing Skill →

Published inTesting

Testing Is a Role, Not a Person

written by

I was going to call this post “Interrogating My Own Code” but that was a bit opaque, even though it’s entirely accurate. I was then going to title it “When the Developer Is the Tester” because that’s part of what I’m talking about. But I ended up on what I believe is the more salient point.

Continue reading Testing Is a Role, Not a Person →

Published inAI

When Performing Causality Means Performing Experience

written by

I originally wanted to title this post “What Is It Like to Be Nothing?” which would be a direct riff on Thomas Nagel’s famous paper “What Is It Like to Be a Bat?” Any philosophically literate reader will recognize that immediately but, in the end, I wanted to continue the theme from my previous post and talk about wider implications.

Continue reading When Performing Causality Means Performing Experience →

Published inAI

When AI Performs Causality Instead of Practicing It

written by

In my previous post, I talked about the cautionary aspect of AI hallucinating its own hallucinations. There’s a deeper element to this and it has to do with causality. Or, rather, the lack of it. In this post, I’ll look at how to test this aspect of AI in very simple way, which will point to a deeper way to consider how to consider the trustability of AI when cause and effect matter.

Continue reading When AI Performs Causality Instead of Practicing It →

Published inAI

When AI Hallucinates Its Own Hallucinations

written by

I recently had an ongoing series on AI and Testing. Today I’m not running evals or measuring correctness scores. Today I’m doing something more uncomfortable: watching an AI fall apart in real time, and narrating what’s actually happening at each step.

Continue reading When AI Hallucinates Its Own Hallucinations →

Published inHistory and Science

Testing: The Art of Unlearning

written by

In two earlier posts I traced the path from Aristotelian to Galilean thinking as a way of understanding how testing developed as a discipline: how competing models of quality, and the slow maturation of experimental method, gave rise to something we might actually recognize as testing today. This post sits in that same current of thought, but takes a step back to ask a prior question: what is it that makes any of that development so difficult in the first place?

Continue reading Testing: The Art of Unlearning →

Published inAI

AI Was Already Here: Loud Opinions vs. Precise Understanding

written by

Previously I had talked about the idea of personal marketability when it came to learning AI. That was in the context of an AI and Testing series, so I trust what was being learned was obvious. I’m seeing an interesting trend developing with ardent opponents of any and all AI, particularly on social media platforms like LinkedIn. This is my attempt at a social response to that.

Continue reading AI Was Already Here: Loud Opinions vs. Precise Understanding →

Published inAIDSPy

DSPy: Declaring Instead of Prompting

written by

In my AI and Testing series, which ran for a couple of months, I focused heavily on the testing side of things. I now want to consider AI in some specific contexts that you are likely to come across and show how those contexts work. The first that I’ll focus on is a tool called DSPy.

Continue reading DSPy: Declaring Instead of Prompting →

Published inAI and Testing

AI and Testing: Evaluation Synthesis

written by

This series has now covered eight metrics across two evaluation paradigms. We’ve applied them to a warp drive paper, a mass extinction paper, a philosophical essay on colliders and cosmology, and an essay on time travel and metaphysics. Each post introduced a metric, showed what it catches, and explained what its scores mean in context. This post does something different: it stops introducing new metrics and instead asks what the full set reveals when used together.

Continue reading AI and Testing: Evaluation Synthesis →

Published inAI and Testing

AI and Testing: Evaluating Conversations

written by

In the previous posts in the DeepEval series, we built up a diagnostic framework for evaluating RAG systems, covering Faithfulness, Contextual Precision, Contextual Recall, Contextual Relevancy, and G-Eval. All of those metrics operate on single turns: one question, one retrieval context, one response, one score. In this post we’ll move into different territory: conversational evaluation.

Continue reading AI and Testing: Evaluating Conversations →

Published inAI and Testing

AI and Testing: Recall, Relevancy, and Richer Evaluation

written by

In the previous posts we looked at the Faithfulness and Contextual Precision metrics with DeepEval, and started building an intuition for how retrieval failures cascade into generation failures. Those two metrics told us what went wrong and where in the pipeline. In this post, we’ll add three more tools to the diagnostic kit: Contextual Recall, Contextual Relevancy, and G-Eval.

Continue reading AI and Testing: Recall, Relevancy, and Richer Evaluation →

Published inAI and Testing

The Last Useful Animal

written by

In my previous posts, I’ve been talking a lot about AI technology and tooling, and any enthusiasm within those posts came from helping people see how and where to test AI, and keep it explainable, interpretable, testable and, thus, ultimately trustable. All that content being said, I have serious concerns about what AI is potentially doing to us: not to our test pipelines, but to us as a civilization. Fair warning: this is a bit of a Thomist indictment of techno-oligarchy.

Continue reading The Last Useful Animal →

Published inAI and Testing

Testing the “Yes-Man” in Your Pocket

written by

If you’ve been following my recent posts on how to test AI, you know that evaluating Large Language Models (LLMs) requires an entirely different mindset than traditional software testing. We’re no longer just testing for crashes, latency, or even factual hallucinations. As AI becomes deeply integrated into our daily lives, we have to start testing for psychological and behavioral impacts.

Continue reading Testing the “Yes-Man” in Your Pocket →

Published inAI and Testing

AI and Testing: From Specification to Story

written by

In the previous post, we built a formal ontology from the Z-Machine specification, used it to drive code generation, and then turned it into a test oracle. At the end of that work, I hinted that the next step might be to put an LLM directly in the loop and watch it actually play one of those games. That’s what this post does.

Continue reading AI and Testing: From Specification to Story →

Published inAI and Testing

AI and Testing: From Ontology to Implementation

written by

In the previous post, we looked at setting up an ontology based on a Z-Machine specification. Our goal was to get this in place so that we could have an LLM generate code to implement the portion of the ontology that we described. In this post, we’ll attempt exactly that.

Continue reading AI and Testing: From Ontology to Implementation →

Published inAI and Testing

AI and Testing: From Specification to Ontology

written by

In the previous posts we looked at setting up a graph pipeline and auditing that pipeline. All of this was based on an ontology, but one that was minimally constructed. Let’s dig more into ontologies here, specifically in relation to an actual specification for some actual software.

Continue reading AI and Testing: From Specification to Ontology →