Voice Agent Stress Suite v1
An open stress suite for voice agents.
Fifty scenarios built to break real-time voice agents where they actually break: noisy rooms, code-switching callers, tangled orders, and people trying to jailbreak the drive-through. Downloadable audio, a fixed rubric, and a scoring script — reproducible, not vibes.
What it is
The Voice Agent Stress Suite is an open, reproducible test set for real-time voice agents. Each scenario ships as an audio clip with a defined caller intent and an expected outcome. You run your agent against the clips, score each turn against the rubric with the scoring script, and publish the numbers — including the ones that look bad. The suite, the rubric, and the script live in one repo so anyone can rerun the exact conditions.
Scenario taxonomy
Fifty scenarios, weighted toward the failure modes that lose orders and callers. Counts below are the suite’s design targets, not results.
| Category | Count | What it stresses |
|---|---|---|
| Acoustic | 8 | Background noise, codec artifacts, and crosstalk — the channel degrades before the words do. |
| Speaker | 8 | Heavy accents, fast talkers, elderly and child voices — one grammar, many mouths. |
| Language | 6 | Mandarin, Spanish, and English↔中文 code-switching mid-sentence. |
| Order complexity | 10 | Multi-item orders, stacked modifiers, ambiguous references, and allergy constraints. |
| Conversational | 10 | Reversals, barge-in, and mid-utterance self-correction. |
| Adversarial | 8 | Prompt injection, profanity, haggling, and “are you a robot”. |
| Total | 50 | Full suite, run end to end per agent. |
Scoring rubric
Every turn is scored 0, 1, or 2 on five dimensions. Two is the ceiling per dimension; the anchors below fix what each point means so two scorers land on the same number.
Ticket accuracy
- 0wrong or missing items
- 1partially correct, one error
- 2exact order, every field right
Clarification appropriateness
- 0guesses or over-asks
- 1asks, but at the wrong moment
- 2asks only when ambiguity is real
Latency within budget
- 0over budget, dead air
- 1borderline or uneven
- 2responds within the turn budget
Containment (no dead end)
- 0loops or strands the caller
- 1recovers, but the caller carries it
- 2keeps a path forward at every turn
Graceful failure
- 0fails silently or falsely confident
- 1fails, hands off without context
- 2names the limit, hands off cleanly
Results
KOTA’s scores publish here once the suite runs — warts included. No results exist yet; the table below shows the structure they land in.
| Category | Ticket accuracy | Clarification | Latency | Containment | Graceful failure |
|---|---|---|---|---|---|
| Acoustic | — | — | — | — | — |
| Speaker | — | — | — | — | — |
| Language | — | — | — | — | — |
| Order complexity | — | — | — | — | — |
| Conversational | — | — | — | — | — |
| Adversarial | — | — | — | — | — |
Submit your own scores
Ran your agent against the suite? Add its column to the public record. Submissions go in as a pull request so the method is visible alongside the numbers.
Clone the repo and run all 50 clips through your agent with the audio and expected outcomes as shipped — no edits to the scenarios. repo: coming
Score with the included script, 0–2 on each of the five rubric dimensions, per category.
Open a pull request that adds your results plus a methodology attestation: agent and version, model and settings, latency budget, date, and anything you changed in the run environment.
Reproducible submissions get merged into the public table. If it cannot be rerun from your attestation, it does not ship.