Putting an AI Agent into Production: The Hard Part Was Never the Prompt

What separates an agent that “runs” from one you “dare to ship” was never a better prompt; it’s an entire stack of evaluation and audit, from the foundation up to the ceiling.

The model will get things wrong. That’s the premise, not the accident. Your job isn’t to keep it from erring; it’s to make sure that when it does, the error lands within a tolerable range.

For ordinary SaaS, drift detection is roughly enough. But the moment you enter a regulated field, a traceable audit trail stops being a nice-to-have and becomes the price of admission.

One day your agent suddenly starts giving wrong answers. You spend hours digging: nobody touched the code, nobody changed the prompt. In the end you find it: the LLM provider silently swapped the model version behind the same API. In a demo this is harmless; a refresh and it’s gone. But in production, especially in fields like investing and finance, it’s an incident.

What separates an agent that “runs” from one you “dare to ship” isn’t a better prompt; it’s an entire system of evaluation and audit. So what does that system actually look like? Below is the layered approach we use in practice, built from the foundation up, one layer at a time.

Layer 0: Tracing / Observability. This is the foundation. Every call’s context, tool calls, reasoning, and final output, all recorded. This layer produces no conclusions on its own, but every evaluation and audit above it grows out of it. If the foundation isn’t solid, everything above is a castle in the air.

Layer 1: Offline Evaluation. Run it before you ship.

• Golden set: working with domain experts, hand-assemble 200 to 500 real scenarios, from input to expected output. After that, every time you change a prompt, swap a model, or touch retrieval, you re-run it. But remember: it isn’t a one-time asset. Real failures caught in production (see Layer 3) have to be continuously categorized and fed back in, or it will always lag real traffic by half a beat and tend to overfit on the old cases you keep re-running.
• Model selection is evaluation: for the same task, run the golden set with models of different tiers and versions, and put the three numbers (score, unit cost, latency) side by side. You pick not the biggest one by default, but the one at the cost-performance inflection point. The expensive model is reserved only for the steps that genuinely demand capability.
• Slice metrics: looking only at the aggregate score will fool you. A system scoring 90 overall might score just 60 on one category of fund, simply averaged up by high scores on other slices. So you must cut the data and look by question type, fund type, time window, and user role at the same time.
• Component-level eval: evaluate each link separately. For retrieval, look at recall@k and precision@k; for tool calls, whether the right tool was picked and the arguments were correct; for the final answer, factuality, citation correctness, output format. Whichever link drops the ball is visible at a glance.
• Refusal calibration: did it actually refuse what it should (e.g. requests for MNPI, out-of-scope actions)? And did it actually answer what it should? A system that refuses everything and a system that answers everything are equally unusable.

Layer 2: Online Evaluation. Validate it in real traffic. Start with a canary: the new version takes only a small slice of real traffic, watch the live metrics (failure categories, user behavior, manual spot checks), and once it holds steady, ramp it up bit by bit.

But the canary is only the first line. What matters more is stacking an independent verifier on top of it: a checker decoupled from the main model that, before a result is returned to the user or before a high-risk action is executed, does three things. One, verify citation: does the cited document actually exist, and did it actually say this? Two, verify response based on citation: is the answer genuinely derived from the citation, or did it make something up beyond the citation? Three, verify whether the question should be answered at all: out-of-scope requests, requests for MNPI, things beyond its capability boundary, blocked outright. It’s independent of the main path, so when the main model gets it wrong, there’s still someone behind it to catch the fall.

Layer 3: Failure Taxonomy. Build a classification for errors. Just knowing “it sometimes gets things wrong” is useless; you have to know how it goes wrong and where to fix it. So lay out the failure modes one by one, then track how the frequency of each changes over time: hallucination (inventing facts that don’t exist), retrieving the wrong document, retrieving the right one but using it wrong, using the wrong tool or passing wrong arguments, right premise but wrong conclusion, format violations, refusing when it shouldn’t or not refusing when it should, getting led astray by crafted input (i.e. prompt injection, where instructions smuggled into retrieved documents or user input hijack it). Without this taxonomy, all you hold is a vague “it’s a bit unstable,” with no idea where to start fixing. And this taxonomy isn’t just for tallying frequencies: from each category, pick representative real failures, clean them into “input to expected output,” and add them to the golden set (back to Layer 1); only then can offline evaluation keep pace with live traffic.

Layer 4: Drift Detection. That scenario from the opening. But the first line of defense actually isn’t detection, it’s locking the version: if you can pin a dated model snapshot, pin it, and bind the model version into provenance (see Layer 5), so the provider can’t silently swap it out without your knowledge. Detection is the backstop, handling the drift you can’t lock down, like a provider’s silent update under the same version number, or changes in your own retrieval corpus. The backstop is to automatically re-run the golden set on a schedule and alert the moment the score drops. But be clear about its blind spot: a daily run only covers the slices the golden set covers; it simply can’t see degradation on long-tail slices. So coverage has to grow alongside live failures; the more complete the golden set, the tighter this line of defense.

Layer 5: Regulatory-Grade Audit Trail. This is the watershed between the investing domain and ordinary SaaS. You have to achieve complete provenance: which prompt version, which model version, which documents were retrieved, which user, what time, with every one of them traceable. When ordinary SaaS has a problem, the most you do is a postmortem; when a regulated domain has a problem, the regulator will make you reconstruct every step as it happened, and if you can’t, that’s on you.

A Few Practical Notes

Boundary design: assume the model will definitely get it wrong. The model is bound to make mistakes; that’s the premise, not the surprise. So when you design, the goal isn’t to keep it from erring, it’s to guarantee that even when it errs, the error lands within an acceptable range. Concretely: cage the actions in a sandbox, tighten permissions, set limits and rollbacks on every tool call, and squeeze the blast radius of an error down to something you can bear.

This is also my answer to prompt injection: you can’t block every crafted input, but you can make “the maximum damage it can do even if the injection succeeds” stay within an acceptable range. That said, the sandbox only contains the action side. Injection can also poison the answer, then use the answer to exfiltrate data; that part has to be caught by the independent verifier upstream (the grounding check plus should-it-be-answered). The sandbox shrinks the blast radius, the verifier catches the poisoned output, and all retrieved content is treated as untrusted data on top. This combination is far more realistic than trying to enumerate every malicious input. At the end of the day, there’s no silver bullet here.

For ordinary SaaS, getting to Layer 4 is basically enough. But once you enter a regulated domain, Layer 5 is no longer a bonus; it’s the price of admission. This whole apparatus looks cumbersome, but its meaning is singular: it lets you dare to leave the agent running, instead of dreading every day how it might go wrong today.