In the previous post we looked at the code for an entire pipeline that uses a lightweight ontology to guide extraction and construct a queryable knowledge graph from unstructured text. Here, we’ll look at auditing what this pipeline is doing.
Continue reading AI and Testing: Auditing a Knowledge Graph Pipeline
In the previous post we talked about the conceptual basis of knowledge graphs and ontologies and pointed toward the code we’ll be using. In this post, we’re going to dive into that code and put the concepts into action.
Continue reading AI and Testing: A Knowledge Graph Pipeline in Practice
If you’ve been following this series, you’ve seen how local LLMs can be used for everything from basic inference to evaluation frameworks. This post takes a different angle. Rather than asking what a model knows, we’re going to ask how we can take what a model reads and turn it into structured, queryable knowledge.
Continue reading AI and Testing: Knowledge Graphs and Ontologies
In the previous post, we looked at a simple web app and looked to see whether a model could generate test cases from the app, analyze the code of that app, and generate automation based on those test cases. Here we’ll refine that process a bit by considering a source of truth and considering different models working together to create a pipeline. We’ll even sneak DeepEval back in.
Continue reading AI and Testing: Using Model Pipelines for Testing
Writing comprehensive test cases means understanding every component, state transition, and edge case in your application. Can an AI model look at a web application and figure out what needs testing? Well, let’s find out. We’ll give a local AI model the HTML for a bomb defusal simulator, ask it to analyze the code, and see if it can generate meaningful test cases, then convert those into working Playwright scripts.
Continue reading AI and Testing: Using Local Models for Testing
We did a lot of testing to determine retrieval quality issues in parts one, two, and three. Here I’m going to close off this particular thread by considering a particular test variation that we have neglected up to this point.
Continue reading AI and Testing: Improving Retrieval Quality, Part 4
In the previous post we ran four experiments attempting to improve our RAG system’s retrieval quality through parameter tuning: smaller chunks, more retrieval, both combined, and semantic chunking. Every experiment either maintained the baseline failure or made it worse. Let’s continue investigating!
Continue reading AI and Testing: Improving Retrieval Quality, Part 3
In the previous post we set up a test experiment around DeepEval and used DeepEval’s evaluation function to establish a quality baseline. That post ended with the need for experiments to confirm against that baseline, and that’s what we’ll do in this post.
Continue reading AI and Testing: Improving Retrieval Quality, Part 2
In the previous post on Contextual Precision, we diagnosed a critical problem in our RAG system: poor retrieval quality was causing failures that we also observed in the Faithfulness post. In this first of three related posts, we’re going to dig in a bit. This will be our first extended example of what testing a generative AI really looks like.
Continue reading AI and Testing: Improving Retrieval Quality, Part 1
In the previous post we looked at the Faithfulness metric with DeepEval and got some intuitions in place about how to start thinking about using metrics in general. In this post, we’ll look at a third metric.
In the previous post we looked at the Answer Relevancy metric with DeepEval and got some intuitions in place about how to start thinking about using metrics in general. In this post, we’ll look at a second metric that requires no faith but is all about being faithful.
In the previous post we got set up with DeepEval. Here we’re going to put that tool to use by looking at our first test case and our first quality metric.
In previous posts in this series, I’ve largely been talking about how to use local LLMs by writing scripts and, along the way, I’ve been able to shoehorn in some testing ideas. We even wrote a bespoke test script together. In this post, I’m going to focus more specifically on testing by considering the idea of evaluation.
In the posts in this series, I’ve been taking you through a lot of concepts and tooling. That’s going to continue but, for this post, it felt prudent to take a little break and talk about why doing all this can matter. That gets into interviewing and potentially being hired.
In the previous post, we refactored a test case that we have been working on. In this post, we’re going to use that refactored test case and scale it up a bit.
In the previous post, we refined an AI test case that we had previously created as a testing example. In this brief post, I want to show a refactoring of that code. We will also align on the output of this test.
In the previous post I provided an extended testing example where we wrote an “AI test case” together. This post will provide some more test thinking around that initial test case.
In this post, my goal is to write a relatively substantive test case and, while doing so, bring together many of the topics talked about in previous posts of this series.
Here I’m going to continue the thread from the previous post, where we started to look at the concept of Runnables, which is really what puts the “Chain” in “LangChain.”
Continue reading AI and Testing: LangChain and Orchestration
In the previous post, we got familiar with LangChain templates and dipped our toes into messages. In this post, I’m going to focus a bit more on those messages since these are the key to communicating with AI.