Pattern library
Patterns for real-time voice agents
The recurring problems in building phone-grade voice AI, and the mechanisms that solve them. Each note frames a problem, its failure modes, and implementation notes drawn from systems that run live on a call.
Barge-in
Treat the interrupt as a state transition: when the caller talks over the agent, cancel and flush in-flight TTS, then resume with context intact.
Read the note →Endpointing and turn detection
Decide when the caller has finished a turn without cutting off slow speakers or adding dead air. Silence timers are a starting point, not the answer.
Read the note →VAD tuning
Setting voice-activity thresholds and hangover windows so background noise does not register as speech and soft talkers do not get dropped.
Read the note →Streaming STT selection
Choosing a streaming speech-to-text path on partial-transcript stability, endpoint signals, and phone-band audio rather than headline accuracy.
Read the note →Menu and domain grounding
Constraining the model to a real menu or catalog so it resolves what the caller said to items that exist, instead of inventing plausible ones.
Read the note →Clarify-before-commit guardrails
Forcing a confirmation step before any irreversible action, so a low-confidence transcript never books, charges, or cancels on its own.
Read the note →Human handoff design
Detecting when the agent is out of its depth and transferring to a person with the transcript and intent carried across the boundary.
Read the note →Telephony failure modes
The failure surface of SIP and Twilio: one-way audio, dropped media, DTMF collisions, and reconnection when a leg goes silent mid-call.
Read the note →Voice eval harness design
Replaying recorded calls and synthetic turns through the pipeline so a change to a prompt or timer is measured, not guessed at.
Read the note →Latency budgets
Accounting for every millisecond from end-of-speech to first audio out across VAD, STT, model, and TTS so the turn feels like a conversation.
Read the note →