Case study
I worked in ops queues for two years. Then I rebuilt the job as an agent — and built the exam that grades it, because an agent you can't measure is just a demo.
At Genpact I worked in a high-volume operations queue: tickets arriving all shift, SLA clocks on every one, root-cause analysis when things repeated, escalation calls when something was above my pay grade. That job is exactly what people now want AI agents to do — and almost every ops-agent project I've seen proves itself with a demo: point it at a few tickets, watch it label them, applaud.
A demo can't answer the questions that decide whether you'd let an agent near a real queue. Does it miss critical tickets? Does it escalate wisely, or does it dump its judgment calls on humans? What happens on the bad days — the surge, the cascade, the shift where the reported severities are wrong? Those are measurement questions, and answering them was the point of building FirstLine.
FirstLine's core is a simulator with hidden ground truth. It generates a realistic eight-hour shift of tickets from fifteen incident archetypes I lived with — cert expiries, bad deploys, DNS failures, payment-gateway timeouts, security breaches — written the way tickets actually read: customers describe symptoms, monitoring quotes metrics. Because the simulator writes each ticket, it knows the true root cause, the true priority, and whether escalation is genuinely required. The agent never sees any of that.
Which means every shift can end in a scorecard instead of an opinion: triage accuracy against truth, priority accuracy, missed P0s, escalation precision and recall, incident-clustering accuracy, SLA breaches — and one composite score where a missed P0 costs ten points, because on a real floor a dropped critical ticket is the fireable offense.
Everything is seeded: same scenario, same seed, byte-identical shift and event log, on any machine. Runs are comparable, bugs are reproducible, and experiments are science instead of anecdotes.
| scenario | backend | triage acc | esc. precision | esc. recall | missed P0 | score |
|---|---|---|---|---|---|---|
| calm_tuesday | rules | 1.000 | 1.000 | 1.000 | 0 | 100.0 |
| fog | rules | 1.000 | 1.000 | 1.000 | 0 | 100.0 |
| calm_tuesday | llama3.2 (3B) | 0.556 | 0.000 | 1.000 | 0 | 70.2 |
| fog | llama3.2 (3B) | 0.857 | 0.529 | 0.900 | 0 | 81.4 |
| mixed_shift | llama3.2 (3B) | 0.564 | 0.212 | 1.000 | 0 | 69.3 |
The model never missed a critical ticket — but only by escalating five tickets for every one that needed a human. Safety by crying wolf, measured.
That's the headline finding, and it's one a demo can never surface. Perfect escalation recall with 21% precision is an agent that is technically safe and operationally useless — it has simply moved the queue onto the senior team, and I've been on the floor where that noise becomes alert fatigue. Three more findings from the same runs:
One honest caveat, stated before anyone asks: the rules baseline scores 100 because it is graded on its home turf — it knows the archetype taxonomy the simulator generates from. It exists as a determinism anchor and an honest ceiling, not a triumph. And the same fog shift, same seed, scores an identical 100.0 on my laptop and on the live deployment — reproducibility, verified across machines.
Multi-seed runs with confidence intervals instead of single-shift cells; few-shot examples per archetype (the model fragments cross-service cascades — cluster accuracy 0.333 on mixed_shift); an escalation budget in the prompt so the agent economises humans; and frontier-model rows in the table alongside the local 3B. Each one is a single eval run away from being a number — that's what the harness is for.
David Jeremie Anand · New Delhi · every number on this page comes from a committed run capture in the repo's docs/runs/.