TL;DR

  • Shipping without proper evals has become standard practice, yet it’s the opposite of engineering

  • Probabilistic foundations mean results will vary: agents will eventually mess up, anywhere from tiny mistakes to completely absurd errors

  • Popular model benchmarks are fundamentally flawed, be very cautious when updating

  • Infrastructure matters: same model on different providers yields different results

  • Build your own domain-specific evals despite imperfections: you control them and get concrete iteration levers

  • Progression is key: start with vibes, then add loose structured tests before getting into layered offline and online evals

  • Five core challenges: Quality (what’s “right”?), Guardrails, Coverage (sample vs everything), Friction (speed vs accuracy), Cost (all of it adds up)

  • There is no perfect solution: combine multiple weak signals (Swiss cheese model)

  • This won’t get easier: evaluation remains messy, imperfect, expensive. It’s the tax for using probabilistic systems with exceptional capabilities

  • Success = adequate defense + speed to iterate when things break

Nothing’s The Same Anymore

Andrew Ng issued a warning a few months ago about how teams are building agentic systems. Instead of endlessly stacking the latest buzzy techniques, he reminded everyone that the real driver of progress is disciplined evals and error analysis. It sounds like common sense. Shipping any piece of code without properly assessing its quality is the opposite of engineering, yet it has somehow become standard practice for many people working with gen-AI.

For anyone who has been living in a cave or slept through the past three years, generative AI is the latest revolution in artificial intelligence, built on probabilistic foundations. This has triggered a dramatic shift away from the deterministic, rigid, and narrow systems we have dealt with since the rise of traditional programming, toward a formidable tool that handles complexity with fluidity and flexibility. It offloads state and control management, the two challenges described in Out of the Tar Pit, from the developer.

I am not going to cover coding agents here, again. If you are reading this, you are probably some kind of engineer assisted by an agent already. You know how useful and irritating they can be. But as a software developer, you at least have the upper hand, since you can review the code, re-prompt, or manually fix things when things go south.

Also, as Ziv Ilan from NVIDIA reminded me during one of our weekly TechLunch, code can be written in many different ways, but the output of our functions is deterministic. Unit/E2E testing, even if you delegate this work to an agent, is still an effective guardrail, especially since it can be incremented at will. Not saying this is easy, but it is solvable.

Making AI Features Work [Most Of The Time]

Therefore, I will focus here on what I believe is the biggest issue: new AI-native products or new AI features implemented directly into the user experience flow. Any kind of copilot writing legal documents, an agent booking a trip that suits one’s vague preferences, generating a fancy picture with basic instructions, a virtual sport coach setting up a program to get back in shape, a synthetic companion, and basically anything where relevance and results are far from deterministic, highly subjective, and difficult, sometimes impossible, to compare. All the reasons why we are very happy to pass the bucket to an AI agent rather than rely on basic algorithms and decision trees.

Probabilistic foundations mean results will vary. The fast pace at which models and tools are released, often without proper evaluation, compounds variance and puts the real difficulty in the spotlight: being able to assess, refine, and detect the happy path (aka the expected scenarios) where the model achieves what you are looking for, and to shrink the volume of sad paths (basically everything else), and to do it efficiently.

Otherwise, a good product that, for some features, depends on models, API providers, MCP servers, and layers of rules and prompts in between, turns very sour very quickly, without even being able to locate where and what went wrong. And when you delegate complex tasks without a clear picture of what the output should look like, how do you assess anything under that level of uncertainty?

This is the big elephant in the room with agents, and gen-AI more broadly. They can be very effective, but there is always a moment when they mess up. It can range from a tiny, harmless mistake to something completely absurd or outrageous one second later. And since there is evidence of a surge of agents in production (23% of organizations are already scaling agentic AI systems in at least one business function, with an additional 39% experimenting with AI agents, according to McKinsey) the entire industry has been scrambling to figure out how to mitigate this.

Ng’s suggestion is to apply, at least, the bare minimum of engineering hygiene: quick prototyping, followed by iterative design of task-specific checks, using tools like LLM-as-judge around concrete failures. It feels more like a bandaid on an open wound at the scale of agent deployments, but it is better than nothing.

Let’s Be Pragmatic

First of all, we will focus exclusively on language models, which is already a complex enough topic. In one of our SOTA webinars published about a year ago, Why Your Evals Are Probably Off? (a must watch!), Ivan P. Yamshchikov from Pleias showed that popular benchmarks such as HellaSwag suffer from fundamental validity issues. In a paper he co-authored (see What the HellaSwag? On the Validity of Common-Sense Reasoning Benchmarks), the team found that models maintained over 65% of their predictions unchanged, simply by selecting longer or more complex-looking answers. HellaSwag is not an isolated case: similar validity issues led to the creation of SWE-bench Verified and WebArena Verified, both built to fix flaws in their original versions (but should still not be considered perfect by any means).

Guillaume Allegre has become an expert on evaluation. He is building Andromede.ai, which focuses on parameterized environments grounded in real-world data, notably websites & tool-use environments, for agent post-training and evaluation. He keeps emphasizing that upgrading to a new model version based solely on official benchmarks, while expecting strictly better results, is risky. Even Yann LeCun has acknowledged that Meta has used “different models for different benchmarks to give better results.”

And more importantly, the model is not always the culprit: far from it. Every layer of the setup can impact outcomes (see “Why benchmarking is hard” from Epoch). And switching from Groq to Fireworks, for instance, while using the exact same model, can lead to very different results, as Tejas Chopra from Netflix’s current study shows, and as he generously shared early results in our Discord community, Reg.exe.

In that sense, Ng’s philosophy is the right one. Everyone should build their own evals - as imperfect as they are - to detect these variances. And because you retain control over them, they give you concrete levers to iterate, improve, or avoid regressions.

How, When And What To Measure

To keep it simple, you have two modes: offline (using static data, like logs or traces after the facts) and online (during live interaction, observing behavior in real-time), each with pros and cons. Each category has its own flavor, again with its own trade-offs. And since you are tech-savvy readers, you can already foresee my conclusion: no solution is perfect. Thank you, see y’all, and good luck with that!

Of course, things are more subtle than that. Even if the idea of perfection left the room a few years ago, some solutions, or combinations of them, can still deliver acceptable results and avoid major pitfalls.

The offline approach is the most accessible, since it puts less pressure on execution time, time you can theoretically use for deeper analysis. On the other hand, the online approach lets you react on the fly and fix things in real time, rather than realizing at the end of the day that your latest update was a huge fuck-up and cost you a lot of credibility with users. But processing time is usually a hard constraint, unless you are fine with adding seconds between calls, so the evaluation tends to be much more superficial.

Now that we have laid out both paradigms, which both are important and complementary, let’s take a step back and list the main challenges of proper evaluation:

  • Quality: Did the model produce the right thing for the user? This might sound easy, and if it is obvious to you at first glance whether the result is right or wrong, chances are you are overengineering with gen-AI where something more trivial or proven, like a logic tree or classic ML, would do a better job at a lower cost. LLMs are usually useful when the outcome is non-binary, subtle, or very complex, with multiple layers and semantics. If you have ever encountered the OpenAI A/B test in ChatGPT asking you to choose between two replies, and felt the effort required to express a preference, you know what I mean. Unless something is obviously off, like a clear hallucination, which you can only detect if you actually know the topic, tuning quality (with little variance) can be genuinely hard.

  • Guardrails: Depending on your industry, this can be one of the biggest liabilities you face. Hijacking a model to repurpose it is, sadly, not rocket science, much like penetrating a company’s network. It can be used either to access sensitive information or to make you pay tokens for someone else’s tasks. Funny enough, this is much closer to manipulation, in the human sense, than to anything remotely resembling programming. It is about going around rules by chaining prompts in ways that slowly lead the agent to reveal things it is not supposed to, like in the famous Gandalf test, where the goal is to extract a password despite increasingly strong protections, or to make it perform actions it should not, such as accessing a free model for someone else’s benefit.

  • Coverage: Are you going to be thorough and review everything, or sample logs from time to time? Things were much easier back then with traditional logs. A few grep commands in highly normalized logs, and you could infer objective reports and build clean dashboards with nice charts. LLM logs look more like a WhatsApp conversation, full of model thoughts, observations, typos, and ambiguity. They are heavy, subtle, and extremely hard to rank or classify. Analyzing everything adds cost. Sampling, or delegating analysis to another model via LLM-as-a-judge, means stacking probability on top of probability. To be fair, it does have benefits, similar to peer review, but it also inherits many of the same flaws: non-determinism, cost, and time.

  • Friction: This mostly applies to online evaluation when you run checks synchronously. In live systems, there is always a trade-off between quality and speed. Take payments as an example. Catching all fraud is critical, but is it better than enabling a much faster payment flow for the vast majority of users, while accepting a tiny amount of fraud?

All the previous choices end up affecting cost efficiency and therefore turn this into a game of trade-offs. Even with offline evals, where latency/friction can be relaxed in exchange for better results, runtime almost always translates into money. Whether you count CPU, GPU, tokens, or human time, cost is the sum of all of it.

These dimensions are so tightly coupled that I doubt there will ever be a perfect solution. But I do believe, at least at the time of writing, that a combination of approaches is necessary, and most importantly, that it should be rolled out gradually.

Vibes Are Your First User

First of all, evals should evolve as the project, its user base, and its activity grow. More importantly, and as obvious as it sounds, they should evolve as you gain clarity on what you are actually optimizing for, product-wise. In that sense, evaluation is so tightly coupled to your vision that it should be treated as your first user. This is even more true given that the very first layer of evaluation is simply you using the product yourself, eating your own dog food, and trying to improve what the agent does with higher probability (called pass@k, the probability of at least one good outcome among k attempts).

Even for accomplished founders like Vincent Nallatamby, who sold his first company to Google, later moved to Google Labs to supervise Google Jules (an autonomous coding agent), and now spends his time building his new ambitious venture in SF, vibe evaluation, meaning judging things yourself, with all the subjectivity that comes with it, is where you start.

You draft a product or a feature that relies on gen-AI, you run it, grimace a bit, then adjust and repeat until it feels like something you would want to see more of. You keep it as basic as possible to iterate quickly. Once again, keep in mind that this stage is much closer to product design and intuition than anything else. There is no data yet, only your perception.

This is an opinionated product or agent development stage. The goal is to identify a plausible customer-problem fit and explore ways to reach problem-solution fit. As the creator, you are batting in the dark until a few good scenarios start to emerge. Not randomly, but without strong signals either. Each back-and-forth helps eliminate obviously broken directions and shape a first, very rough evaluation, mainly to catch unequivocal sad paths.

Differentiate North From South

At this point, the goal is simply to get a sense of the general behavior of your agent. You are not aiming for precision yet, but for a rough alignment between your vision and most outputs, with a big emphasis on “most.” You are still in the jungle, trying to figure out how to tell north from south.

After a few steps in the right direction, and hopefully a few dozen (alpha) users, it is time to level up and start objectifying the tests. One approach is to list a few hundred prompts, along with roughly expected outputs. Again, do not overinvest here, and obviously use an LLM for this task, with loose testing. For instance, if you work with structured output, just check that the five required fields are returned. Or verify that a specific term appears in the reply. In other words, avoid rigidity. You are still in the exploratory phase, and you do not want to spend more time fixing the tests than building the feature itself.

The goal is to filter out obviously bad outputs with a bit more structure and refinement, not to lock down the product. Chances are your vision is still evolving. LLM-as-a-judge approaches are quite handy at this stage, or, if you have a lot of money and time, a few human domain experts reviewing results. And as the product and user base grow, the number of input-output combinations explodes, and that is where the real problems begin.

Scaling: The Uncharted Territory

Nallatamby observed that things start to get tough once you cross roughly 10k users. A point where you start to get noticed by a more demanding audience asking for more than a garage prototype, and can no longer rely on vibes and crude, homemade evals based on a few hundred tests, as usage is likely broader than what you initially expected. Your evals are good enough to produce something usable most of the time, and within the perimeter of your initial vision. But with that many users, usage becomes dense and introduces a variety you probably did not anticipate, which can be a good thing, by the way. People try things you did not foresee, and they expect more from you as you grow as a brand, or simply because they start to rely on you.

You could say that the biggest models in the field, like ChatGPT, are still prone to hallucinations. But even if we’ve gotten used to cutting general-purpose tools or raw foundation models some slack, the bar is much higher for your product. You are supposed to provide the layer in between: a specialized service people pay for. As a result, your customers implicitly expect something close to deterministic output.

The ChatGPT homepage says “What can I help with?”, which is deliberately open-ended, not “We will parse professional documents, and the revenue of your company will depend on our ability to do it right. #trustmebro”

Some Ingredients (Mix Em Wisely)

Even if everyone acknowledges that there is, so far, no optimal solution, the answer looks more like a recipe. Ilan summarizes it pretty well: “You have a few different approaches. You could use synthetic data generation or your own data to create these use cases, case studies. Also, if it happens that there are benchmarks out in the wild that fit what you are trying to measure, it can be useful, and you have human evals. Online, you could still use A/B testing.” But his key point is “it’s all a combination, and you run it all, all the time.” All of these tests come with biases and flaws. As a result, his recommendation is to combine multiple weak signals, since no single approach works on its own. Think of it as a Swiss cheese model of defence.

Quentin Churet from Alpic confirms how things evolve with product maturity: “The first time you release an agent, LLMs already bring value without rigorous testing, and your priority is collecting data about what people actually ask. But once you start developing more complex systems or updating your agents (adding tools, changing prompts, switching foundation models) you need strong tests. Otherwise it’s a gut feeling, and you might break what works without knowing it.”

The difference with the old world is that people’s behavior is not measured through basic signals like views, scrolls, or clicks, but through intent expressed via prompts. And the nature of prompts is rarely a single question leading to a single answer. More often, it looks like a full conversation (between human and AI but also with AI talking to AI in autonomous flow), followed by an expected outcome where no ground truth exists. Single-turn tests therefore miss most context-dependent behavior.

Anyway, the workflow he recommends sounds familiar: record sessions, use LLM-as-judge to grade flows, transform good examples into offline test datasets. Some teams then take this further with tools like GEPA (see the research paper) to automatically optimize prompts based on evaluation feedback, though this adds another layer of AI-as-judge costs (along with uncertainty about future model availability). With a critical insight: “It should actually be the product teams, the people who know the task the agent is supposed to do, who review this, rather than just engineers.” For Churet, tools like Braintrust make this “product-friendly” enough.

None of this is rocket science, but it takes time and offers no guarantees. And test sets decay as products evolve. Even small pivots, sometimes just an inch, can invalidate entire test suites.

Now, with a small disclaimer on bias (I am an investor in both companies), let me show two very different philosophies for approaching evaluation:

Orq, the One-Stop Platform to Rule Them All

We have talked a lot about how to progressively add layers, precision or rigidity to your workflow, but this is still a significant amount of work, time, and effort to put together. That is why Sohrab Hosseini and Anthony Diaz built Orq.ai. In their vision, AI agents require an end-to-end lifecycle with equivalent depth (if not more) than traditional software development, and this lifecycle should be delivered as a single platform rather than, as most teams do today, by assembling layers of APIs, open-source packages, and homemade code as features or glue.

Hosseini and Diaz were fully aware that many excellent tools already exist, each strong within its own scope: OpenRouter, LangSmith, Langfuse, Arize AI, LangGraph, you name it, and a lot of them include some eval systems.

But as confirmed by the Epoch article mentioned earlier, each of these services comes with its own assumptions, philosophy, and, more problematically, its own lifecycle. The resulting scaffold becomes fragile quickly, as every layer evolves independently, adopts new patterns, and moves at its own pace, especially since most are either open-source projects or startups still converging on product–market fit.

First, this makes regression root-cause detection significantly harder. More fundamentally, for Hosseini, evaluation is not about having the best individual tools, but about enabling tight collaboration across layers: routing models based on environment or rules, turning specific evals into blocking guardrails, automatically feeding real production edge cases back into datasets when evals trigger, running evals asynchronously on traces at a chosen sampling rate, applying different policies per use case or per evaluation, and so on (see a few demos here). This level of integration is extremely hard to achieve smoothly with a stitched-together stack, regardless of how strong the individual services are.

Moreover, evaluation is not reserved for technical profiles. It needs to be usable by people with no engineering background. Orq reflects this by offering playgrounds that allow non-technical users to try new models, work on prompt engineering, leverage one of the 70–80 ready-to-use evals available in a hub as a starting point, analyze and annotate outputs, and run online or offline evals without touching code.

Moyai, Aiming For Highly Efficient Online Evals At Scale

Robert Hommes shares Hosseini and Diaz’s belief that pre-AI platforms are unable to adapt to, or solve the issues raised by, this new paradigm. Over the past few months, he has been building a highly efficient online evaluation that is both cost- and speed-efficient, without compromising on quality. More importantly, it can run at scale without sampling, meaning it can process 100% of the logs, and it is already starting to show promising results.

His startup, Moyai, is building agent behavior monitoring for production and detects anomalies before your users do. To him, existing observability platforms are mostly rebranding themselves for agents without actually solving the new problems introduced by probabilistic, and unfit for multi-step, and multi-tool systems. And as already mentioned, using AI-as-judge at scale is both expensive and biased, models tend to prefer outputs from their own family.

Moyai’s approach is to parse all logs, group them into “prototypes” based on structural patterns, and detect obvious errors on the fly at very low cost, running on CPU, without tokens involved, and with sub-millisecond latency. It could be something as basic as: search_database → 3 results found → synthesize_response → user_satisfied. A failure pattern: search_database → 0 results → fallback_attempt → timeout → error. These execution sequences group similar flows for comparison without analyzing content.

Then, within each prototype stream, Moyai performs semantic detection using embeddings, which remains very cheap, and applies expensive AI-as-judge evaluation only to the small number of anomalous cases to determine whether they are actual issues.

The key insight here is rebalancing. Hommes explained the false-positive problem clearly: when 95% of logs are fine, even a reasonably accurate detector will generate more false positives than true positives.

Let me explain. A 90% detection rate sounds amazing on paper, but remember that this applies to all logs. That means that while it can catch 90% of bad scenarios, it will also misclassify 10% of happy paths as errors.

For instance, imagine a dataset of 10,000 logs, where you know for sure that 95% of operations went well and 5% went wrong, which represents 500 errors total. Running a detector with a 90% success rate will produce:

  • 450 true positives, 90% of the 500 bad logs, which is the goal.

  • 950 false positives, 10% of the 9,500 good logs, where valid scenarios are flagged as errors.

You end up with 1,400 alerts, but only 32% of them correspond to real problems.

The human feedback loop completes this progressive filtering: when anomalies are flagged, engineers review them, categorizing each as a true or false positive. This feedback reinforces detection patterns and further reduces outliers over time.

Moyai has now been running for weeks on a fairly heavy production workload with strong results, improving quickly, and could be a compelling solution for large-scale log analysis to evolve the system guarantees that can be given on your AI agents.

The Uncomfortable Truth

Here’s what nobody wants to hear: this isn’t getting easier anytime soon. The probabilistic nature of LLMs means evaluation will remain messy, imperfect, and expensive. Even if things still feel chaotic, as models evolve, tools like Orq and Moyai, frameworks like GEPA, and overall methodology will adapt and help us tame this technology.

The potential is so large that the industry is already shipping agents at scale while figuring things out along the way, sometimes feeling a bit like we’re Gromit laying the tracks just ahead of a train moving at full speed. And while evaluation frameworks are improving, they are essentially sophisticated damage control.

This is probably the tax we all have to pay for using probabilistic systems with exceptional capabilities. If you want the flexibility and power of LLMs to handle complexity that traditional code cannot, you accept the evaluation burden. There is no free lunch.

The companies that will succeed are not the ones with perfect evals, since those do not exist. They are the ones that manage to build the right level of defense while still moving fast enough to iterate when things inevitably break.

Andrew Ng is right. Disciplined evals and error analysis are the real drivers of progress. But let’s be honest about what disciplined means in this context. It is not the rigorous, deterministic testing we are used to. It is a continuous, messy process of measuring, learning, and adapting as both your product and the underlying models evolve.

Now, go fix your evals before your users do it for you.

❤️ Thanks to Guillaume Allegre, Robert Hommes, Quentin Churet for the solid contribution and proofreading!

🙏 Memes stolen from here, here, and here