An AI assistant gives the wrong answer. You change one prompt, test it again, and it gives the right answer.
Problem solved?
Not necessarily.
That is one of the realities of building AI-powered products. Getting a model to produce a good answer is only part of the work. Teams also need to understand why it produced that answer, whether a change actually improved the system, and what happens when that same system encounters hundreds of slightly different cases.
This was the focus of a recent internal Galeyo workshop on evaluation and tracing with LangSmith. The session looked at a practical question that becomes increasingly important as AI moves from prototypes into real products: how do you know when you can actually trust the behaviour of your system?
One of the examples used during the workshop was a simplified insurance assistant.
A user asks a question and the application first decides whether it relates to home or car insurance. It then searches the relevant data, retrieves the useful information and sends that context to the model before generating a final answer.
The workflow sounds straightforward until a question falls somewhere between two categories.
In one example, something had been stolen from a car parked at a home. The system classified the question as an auto insurance case. At first glance, that made sense. But in this particular scenario, the relevant coverage belonged to home insurance.
The interesting part was not simply that the final answer was wrong. It was being able to see where the mistake happened.
By looking at the trace, the team could inspect the classification step and see that the system had focused on the fact that the item had been stolen from a car. It had not given enough weight to the location where the incident happened.
That changes the debugging process completely.
Instead of asking why “the AI” made a mistake, the team can identify the exact step where the behaviour changed and work on that part of the system.
This is especially important because an AI application is rarely just one prompt and one response. There may be classification, retrieval, business logic, several model calls and multiple intermediate steps before the user sees anything.
Tracing makes those steps visible. It allows teams to inspect the original input, prompts, retrieved documents, intermediate outputs, final responses, execution time and token usage in one place.
Once a system grows beyond something one developer can keep entirely in their head, that visibility becomes increasingly valuable.
After identifying the classification issue, the prompt could be adjusted to take additional details such as location into account.
The same example was tested again and the result was correct.
But that still does not prove the problem has been solved.
Generative AI systems do not behave like traditional deterministic software. A successful response in one test does not guarantee the same behaviour across repeated runs or slightly different inputs.
This is where evaluation becomes important.
Instead of testing only the case that originally failed, teams can create a reference dataset with examples where the expected behaviour is already known. In the insurance example, that could include questions that should clearly be classified as home insurance, others that should go to auto insurance, and more difficult cases where several details influence the correct result.
Every time a prompt, model or part of the workflow changes, the same dataset can be run again.
That makes it possible to see whether the original problem improved without introducing new ones elsewhere. It also makes changes easier to compare over time, including differences in accuracy, latency and token usage.
In practice, this brings AI development closer to regression testing.
The question is no longer whether one example works after a change. The question is whether the system still behaves as expected across a representative set of cases.
Another example from the workshop focused on hallucination.
The model produced a specific numerical answer even though that number was not present in the documents available to the system.
The response sounded convincing and it answered the question directly. But the information had not come from the source the application was supposed to use.
That distinction matters.
For retrieval-based AI systems, it is not enough to evaluate whether an answer sounds correct or whether it addresses the user's question. Teams also need to know whether the answer is grounded in the information the system actually retrieved.
A model may produce a relevant answer and still introduce unsupported information.
That is why different types of evaluation matter. One evaluator might measure whether the answer matches the expected result, while another checks whether the response can actually be supported by the provided context.
Looking at only one of those signals can create a false sense of confidence.
Testing before release is only one part of the picture.
Real users will ask questions nobody included in the original dataset. They will phrase familiar questions differently, combine topics in unexpected ways and expose edge cases that controlled testing did not anticipate.
Production traces can therefore become an important source of new evaluation data.
Teams can monitor interactions, flag suspicious results, review cases with poor retrieval scores and connect user feedback with the trace that produced a particular answer. Selected production interactions can then be reviewed and added to the evaluation dataset.
Over time, the test set becomes more representative of how people actually use the product.
This also makes human review more focused.
The goal is not to have someone manually inspect every interaction. Automated evaluation can help identify which cases deserve attention, while people step in where expert judgement is actually needed.
The system handles scale. People handle the cases where context and judgement matter most.
LangSmith was the tool explored during this particular workshop, but the underlying idea is broader than one platform.
When teams build AI into real products, they need to be able to understand what happens between the user's input and the final answer.
They need to know what information was retrieved, what the model received, which part of the workflow made a decision and whether a change improved one behaviour while making another worse.
At Galeyo, we do not see AI implementation as simply connecting a model to an application.
The model is only one part of the product. The workflow around it, the data it can access, the retrieval logic, prompts, evaluation, monitoring and human review all influence how reliable the final experience becomes.
AI will not become perfectly predictable. But it can become much easier to observe, test and improve.
And that is the difference between an AI feature that works in a demo and an AI product a team can continue building with confidence.
At Galeyo, we help teams move AI beyond experimentation and into products that are built to perform in real-world conditions. From system evaluation and AI integration to custom software development, the goal is the same: build technology that can be understood, improved and trusted as it evolves.
Building an AI-powered product or looking to make an existing one more reliable? Explore how we Develop & Implement AI-powered solutions, or get in touch with Galeyo to tell us what you’re working on.