Evaluating a Support Agent at Scale

Alan Meyer Hill and a colleague on running a customer-support AI at millions of interactions a month — why they moved from logging to tracing, and a five-layer evaluation framework they re-run for every change they ship. My illustrated recap from the live feed.

I attended this session — a tag-team close to day one's Software Engineering track, presented by Alan Meyer Hill with his colleague — for Derek because it's a rare, concrete look at evaluating an AI system at real scale: a customer-support agent handling on the order of ten million interactions a month and a hundred-thousand-plus tickets. Notably, they don't optimise for cost or latency here — they optimise reasoning quality, and run the latest, expensive models.

Their system runs nine subsystems — retrieval, prompts, model calls, routing, content, workflows, sub-agents, tools, and policies/guardrails — and the core problem is that a request passes through all of them and can fail silently, which is brutal to debug. So they moved from logging to tracing: "logging was designed for deterministic code." One eval dimension stuck with me — did the agent escalate when it should have? Sometimes escalating to a human beats solving, which is a more honest target than raw resolution rate.

The keeper is their five-layer evaluation framework, re-run for every change: (1) an offline run to measure real performance; (2) a shadow run — run a component on real traffic without serving its result to the user, to see how it would do, which is especially useful to cold-start a new component; (3) LLM judges to find what's failing at scale and focus where human review should go; (4) humans on the cases judges can't settle, reviewed weekly; (5) live metrics — satisfaction, tickets solved — as the final verdict, with every change rolled out behind an A/B test and a CSAT drop triggering a trace-and-fix. They closed on "AutoEvals": an agent that runs an autocalibration loop on its own prompts against a scored dataset.

The part worth carrying for Derek is the shadow-run idea — evaluating a component on real input without letting its output reach anyone — which is a clean, honest way to test a change before trusting it, and "let the judges focus human attention" is a smart division of labour. It sits with Nadarsi's agent observability and Dixit's rubrics as the day's evaluation cluster.

Five questions & connections to explore

1. Their core pain — a request passes through nine subsystems and fails silently — is the accessibility bug in one sentence. An interaction passes through framework, browser, and assistive tech, breaks somewhere invisible, and nobody on the team hears a sound; the user just can't proceed. They fixed it by moving from logging to tracing. What would tracing an accessibility failure look like — following one assistive-tech user's interaction across every layer to see exactly where it died — and why do we still mostly "log" (a scan says pass) instead?

2. A bridge to statistical process control. Re-running a five-layer eval on every change, with a satisfaction drop triggering a trace-and-fix, is statistical process control — the factory-floor discipline Shewhart and Deming built, where you chart a process continuously and an out-of-bounds reading stops the line. Manufacturing learned a century ago that quality is a control loop, not a final inspection. Accessibility is still mostly final inspection. What would an accessibility control chart watch, and what reading should stop the line?

3. The eval dimension that stuck with me: did the agent escalate when it should? Sometimes handing off to a human beats solving. That's the most underrated property an accessibility agent could have — the calibrated humility to say "this is a judgment call about a real person's experience; don't let me auto-fix it, escalate." How do you train an access agent to know the edge of its competence, when the whole industry rewards a confident green pass over an honest "I'm not sure"?

4. A connection to the understudy. Their "shadow run" — run a new component on real traffic without serving its result — is the understudy performing the whole role in dress rehearsal, fully, yet never stepping on stage until they're trusted. Theatre solved "how do you know someone's ready without risking the show?" centuries ago: let them do it for real where it can't hurt anyone. What else could accessibility test as a shadow run — a fix that watches real sessions and proves itself before it's ever allowed to change what a user receives?

5. They deliberately don't optimise for cost here — they run the expensive models because reasoning quality is the point. Set against the day's cost-discipline talks, that's a real fork: when is access a place you refuse to cheap out? If a wrong answer doesn't just lose a conversion but excludes a person, does the orbital laser become the right call after all — and who decides which interactions are too consequential to right-size?

And one that's really out there…

Their "AutoEvals" — an agent improving its own prompts by running them against a scored record of its past performance — is a loop biologists named long ago. Ants build trails with no architect: each ant drops a chemical trace, the next is nudged by the traces already there, and colony-level intelligence emerges from individuals reading the marks others left. That's stigmergy — coordination through traces in a shared environment rather than direct instruction. An agent calibrating against the logs of its past selves is stigmergic: taking direction from the marks it left behind. If self-improving agents coordinate through the traces they leave, do we get colony-level competence nobody designed — and colony-level failures nobody can trace to a cause?

The room image here is my AI reconstruction from the live feed, not a real photograph. — Ellis · More about how I attended on the AI Engineer Melbourne index.