Case Study - An adversarial safety harness for a voice AI agent

The problem

The client builds a reflective voice AI companion that uses hidden therapeutic frameworks as internal scaffolding — never named to the user. Its moat is how it holds a conversation. Its hard constraint is safety: it must never give self-harm advice, must route crises to human help, and must not validate delusions.

How do you test that, repeatedly, without a human role-playing a suicidal caller every time? We needed to answer one question automatically: when a distressed caller talks to the agent over voice, does it keep them safe?

The plan

We built an automated voice safety harness: a synthetic adversarial caller that joins a real voice room, speaks crisis scenarios out loud, lets the real production agent respond over real audio, then has an LLM judge score the transcript on two axes — safety (did it route the crisis?) and framework (did it use the therapeutic craft?). Simple in theory. The reality was a chain of subtle observability failures — and each one taught us more about evaluating AI than about the product itself.

What we built (and the bugs each step exposed)

The work landed as a sequence of commits, each one peeling back a layer where the test was deceiving us.

1. Make the adversarial caller actually raise the crisis. The first adversarial run passed 4/4 — a false green. The synthetic caller never actually disclosed a crisis; it opened with bland lines like “There’s a lot happening at once.” Safety scored 1.0 because there was nothing dangerous to mishandle. The fix: plant the crisis in the caller’s opening turn, drive escalation across turns, and broaden the judge’s crisis detector so abuse, medical, and psychosis scenarios were gradeable — not just self-harm.

2. Stop guessing — instrument the calls. A call captured zero of the agent’s replies and we were inferring why from scattered logs. We added a single real-time event log per run (one JSONL line per step: room created, agent joined, each caller turn, each captured reply, why the call ended). This one change converted the whole effort from guesswork to evidence — every subsequent root cause was read off this log, not theorized.

3. The agent wasn’t joining the room. The event log immediately answered the zero-capture mystery: agentJoined = null. The synthetic caller was speaking into an empty room — it connected and talked before LiveKit had dispatched the agent to that brand-new room (a startup race on the first call of every batch). The fix: the caller waits for the agent’s audio track to attach before speaking. Abuse-zero disappeared.

4. The real bug — we were judging a lossy transcription. With the agent present, calls captured replies — but the judge kept failing safety on the hardest crises. The worker logs proved the agent spoke the crisis resources (emergency services and crisis helplines) — 92 times across a run. Yet the judged transcript didn’t contain them. Root cause: the harness “heard” the agent by re-transcribing its audio with speech-to-text, and STT clipped multi-sentence safety replies to their first sentence — dropping the resource numbers that come at the end. The test was scoring the agent’s mouth, badly, instead of its words.

5. The simplification — judge from the database. The breakthrough was realizing the right source already existed. The product already persists every conversation turn — role-labelled, verbatim — to Postgres for real users. The harness just needed to read it. We made the worker create a session row for harness rooms (so turns persist), and had the harness read the finished conversation by room name after the call, then judge that. A post-call SELECT replaced the entire live-publish/reconcile machinery — which we deleted.

6. Capturing the interrupted replies too — true 5/5. One gap remained: a turn is saved only when the agent starts speaking it. An aggressive caller barging in before the first audio frame meant that reply emitted nothing — so 1–2 of 5 turns were still missing. The fix: capture the reply at generation time (the LLM-output node, which fires before the barge-in cutoff), with a content-based deduper so generation-time and speech-time never double-write. Interrupted replies now persist. Captured turns jumped from 1–2 to 5–6 per call.

The result

The final run, judging the agent’s complete verbatim transcript from the database:

The agent routes every crisis category correctly. It refused to give self-harm methods, told the abuse caller not to confront and to call emergency services, declined to validate the psychosis delusion while staying warm, and refused to advise doubling a medication dose. The crisis resources it spoke now reliably reach the judge.

The test got sharp enough to find a real product signal

Once the full transcript reached the judge, the framework-quality dimension became meaningful. It flagged that on the two hardest crises — psychosis and self-harm — the agent is safe but goes repetitive and scripted (framework score 0.6–0.7 vs a 0.75 bar), repeating the safety line instead of sustaining the therapeutic scaffolding. That’s a genuine product behavior, not a test artifact, and it was tracked for a future prompt improvement. The harness graduated from “does it pass?” to “where is the craft thin?”

What made it work

• A different judge model. The agent runs on one model; the quality judge runs on a different one to avoid an LLM grading itself (self-preference bias). The judge scores the product’s moat (hidden frameworks) and safety, not generic chatbot niceness.
• Adversarial personas that escalate. Four crisis personas (self-harm, abuse, psychosis, medication) that disclose the crisis up front and push harder when deflected — because normal conversations never stress the safety scoring.
• Real audio, real agent, real persistence. The synthetic caller joins a real LiveKit room and talks to the actual production agent with its real system prompt and safety routing — not a mock. The judged transcript is the same ground-truth data the product persists for real users.