Demos prove the agent shipped. Dashboards prove the agent is running. Only an eval harness proves the agent is working.
Most production agents in 2026 don’t have one. That’s the failure mode I see at scale, and it’s the failure mode that compounds quietly until the operational debt is structural. This article is the long-form version of the argument I’ve been making across client engagements through 2025-2026: if you’re shipping AI without evals, you’re not shipping a system. You’re shipping a long-form demo that costs production money.
What “the agent is working” actually means
Three different statements get conflated:
- The agent shipped. It’s deployed. It runs without crashing. The demo recorded looks good.
- The agent is running. It’s executing in production. Logs exist. Throughput numbers are visible.
- The agent is working. It’s making correct decisions on production data, with calibrated confidence, at acceptable latency, with a known failure-mode taxonomy.
Most teams in 2026 conflate the first two with the third. They have shipped + running. They claim “working” — and they have no artifact that supports the claim.
The eval harness is that artifact. It’s the only artifact. Logs and dashboards tell you the agent is running. Customer feedback, when it arrives, tells you the agent is failing. Neither tells you the agent is working between those two states.
The five-component eval harness
Repeating this from the POV stream because it’s the load-bearing definition. A real eval harness has all five components. Missing any one is a system you can’t trust.
1. Held-out cohort
Real production data the agent has never seen during prompt design, fine-tuning, or training. Refreshed quarterly. The cohort’s outcomes are labeled — by humans, by the underlying ground truth (close-won/lost, replied/not, accepted/rejected), or by a downstream system.
The held-out cohort is the only data on which the agent’s score translates to real-world performance. Performance on the data the agent was prompted on is a leakage check, not a capability check.
2. Scheduled weekly run
The eval runs every week on the same day, automatically. The output goes to a Slack channel or a dashboard the team actually reads. Hard rule: if the eval doesn’t run, the agent doesn’t ship to that week’s cohort.
This is the discipline most teams skip. They build the harness as a one-shot script the team runs “when they think about it.” That’s not a harness; that’s a checklist someone forgets. The schedule is what makes the harness load-bearing.
3. Multi-metric output: precision, recall, calibration
Not just one number. Precision tells you how often the agent’s positive predictions are right. Recall tells you how often the agent finds the positives that exist. Calibration tells you whether the agent’s confidence still tracks reality — and is the metric most teams underweight.
A scoring agent that was 95% confident in January and 95% confident in May with very different precision underneath is the silent killer. The agent’s confidence didn’t change; its actual performance did. Dashboards show the confidence; only the eval shows the underlying calibration.
4. Latency tracking
P50, P95, P99 latency on agent decisions. An agent that’s accurate but takes 90 seconds per decision is fine for 100 leads/day, catastrophic at 10,000. Production scale changes the latency requirement; the harness needs to track it as accuracy does.
Latency drift often signals an underlying problem the team would otherwise miss — model degradation, tool-call timeouts, retry loops, downstream system pressure. The eval surfaces the drift before the throughput problem becomes a production incident.
5. Failure-mode taxonomy
When the agent gets it wrong, why? Categorized. Trended over time.
Common categories I see in scoring/drafting agents:
- Stale-data error — agent acted on data that was 48+ hours out of date
- Ambiguous-input error — input was missing context the agent reasonably needed
- Model-overconfidence error — agent’s confidence was high but the prediction was wrong
- Missing-context error — relevant context existed but wasn’t surfaced to the agent
- Hallucination — agent asserted facts not supported by available data
- Conflict-with-truth — agent’s output contradicted explicit CRM/system state
- Prompt-injection failure — agent followed an instruction in user-provided text rather than the system prompt
The taxonomy gets sharper over time. Trends tell you which category is growing — which is exactly the signal you need to know what to fix next. Without this, every failure feels novel and the team plays whack-a-mole on patches.
What people get wrong about evals
Three common mistakes worth naming.
Mistake 1: Treating evals as model evaluation
A lot of “evals” in 2024-2025 were really model benchmarks — running the model on standard test sets and comparing scores. That’s a different thing.
A production eval is testing the system: the agent’s prompt, the tool surface, the retrieved context, the decision logic, and the model. The model is one input. The system’s behavior on production data is what matters.
Teams that confuse model evals with system evals get false comfort from rising benchmark scores while their production system gets quietly worse.
Mistake 2: Evaluating against synthetic data
A team builds an eval against synthetic test cases the team made up. The agent passes the eval. The agent fails in production.
Synthetic cohorts can supplement, but they cannot replace, real-data cohorts. Production data has tail behaviors the team didn’t think to write down. The held-out real cohort is what surfaces them.
Mistake 3: Running the eval but not acting on it
The eval runs. The Slack message lands. Nobody reads it. The agent’s calibration drift continues for three months. The team only acts when a customer complaint arrives.
The eval is a feedback loop. If the loop isn’t closed — if the team doesn’t have a standing meeting where the eval results are reviewed and trigger action — the harness is decorative. Every week without a closed loop is a week of unacted-on signal.
When the eval saves the agent
Two patterns I’ve seen repeat.
Pattern A: Calibration drift caught at week 14.
A scoring agent shipped in production, performance held for 12 weeks, then calibration started drifting around week 13. The eval caught it. The team’s investigation traced the drift to a CRM schema change upstream that subtly changed the data shape the agent was reasoning over. Fix took two days. Without the eval, the drift would have continued for ~6 weeks before customer-side complaints surfaced — by which point the SDR team would have been working off bad scores at scale.
Pattern B: Failure-mode shift at the model upgrade.
A drafting agent migrated from Claude 4.5 to Claude 4.6. Aggregate accuracy improved on the eval. The failure-mode taxonomy showed a new category emerging: model-overconfidence-on-edge-cases. The eval surfaced it inside one week. The fix was a prompt adjustment that prevented two weeks of drafting confidently-wrong copy into production.
In both cases, the eval did its job: it surfaced a problem before customers did. That’s the entire economic case.
What “good enough” evals look like in 2026
A team without much eval infrastructure can stand up a serviceable harness in a week:
- Day 1 — define the held-out cohort. Pull 200-500 real records the agent has not seen. Label them (by hand if needed, or pull from existing ground truth). Store in Supabase or a CSV in version control.
- Day 2 — build the eval runner. Calls the production agent on each cohort record. Computes precision, recall, calibration buckets, latency. Outputs JSON.
- Day 3 — wire scheduling. Cron job (Supabase Edge Function, GitHub Actions, n8n) that runs weekly. Posts the report to Slack with a stable schema.
- Day 4 — set up the failure-mode taxonomy. Manual labeling of the first ~50 failures. The taxonomy starts coarse and sharpens over time.
- Day 5 — establish the standing meeting. 30 min weekly. Three people minimum: the agent owner, a domain expert, an engineer. Review the report. Decide what to fix.
That’s it. The harness is now load-bearing.
The team that did this in week one of agent deployment is months ahead of the team that’s planning to “get to it after we ship” — because the team without the eval is, by definition, not going to know when the agent silently breaks.
The CODN angle
By the time a team without evals notices that the agent is failing, the agent has been failing for a long time. The failure modes I see most:
- Scoring agent disqualifying high-fit leads. SDR team works smaller cohort. Pipeline misses by a noticeable but unattributed margin. Diagnosis comes after a month of soft pipeline. By that point the team has missed ~20-30% of the qualified leads it should have worked.
- Drafting agent producing hallucinated stats. Goes out under the company’s name on LinkedIn / outbound / blog. The reputational damage compounds with audience size. By the time someone fact-checks the agent’s claims, hundreds of pieces are out.
- Routing agent sending Tier 1 to slow queues. Best opportunities are also the slowest to respond. Win rates drop on the highest-value cohort. Diagnosis usually comes from a deal-review postmortem, by which point five or six six-figure opportunities are dead.
The CODN of skipping evals is not the eval cost. It’s every silent failure between the agent’s deployment and the day someone finally measures.
A working eval harness in week one prevents all three. A missing eval harness invites all three — at compounding scale.
The bottom line
Demos prove the agent shipped. Dashboards prove it’s running. Only the eval proves it’s working.
If you’re shipping a production agent in 2026 without an eval harness, you are not shipping a system. You are shipping a long-form demo, paying production money for it, and accumulating silent failure debt against the day someone finally notices.
Evals — or it’s vibes. The harness is a week of work. The cost of skipping it is everything the agent silently gets wrong between deployment and the day you finally measure.
Build it before you ship the next agent. Retrofit it on the agents already in production. The harness is the only artifact that proves AI works — and the only one that compounds in your favor over time.