What the canary measures
| Metric | How it is measured | What it tells you |
|---|---|---|
| Time to first audio | From the last non-silent frame of the canary's utterance to the first decoded audio frame from the agent | The silence the caller actually hears, per region |
| Turn latency p95 | Across all turns of the scenario, per run and over time | Regression after a model, prompt or SDK change |
| Barge-in stop time | The canary speaks while the agent is talking; time until the agent's audio stops | Endpointing and interruption handling |
| Scenario result | Each turn's reply checked against expectations: contains, regex, DTMF, transfer, intent via rubric | Whether the flow still works, not only whether it answers |
| Task completion | Reached the scenario's end state (confirmation, hand-off, hang-up) within the turn budget | The outcome a product owner cares about |
| Post-dial delay | From INVITE or dial to the first ringing or answer | Carrier and trunk health on the phone leg |
| Media leg | RTT, jitter, loss, concealment and an E-model MOS from getStats() or RTCP | Whether a slow reply is the network or the agent |
| Pipeline stages | End of utterance, time to first token, time to first byte of TTS, when your framework posts them | Which stage of the agent moved |
Scenarios, not pings
A scenario is a scripted dialogue with expectations per turn. Deterministic checks run first; a rubric-based judge is optional, and its output is stored with the run so that a disputed verdict can be read, not guessed.
- id: support-change-email
type: voice_agent
regions: [us-east, eu-west]
interval: 300s
target: { kind: phone, number: "+1xxxxxxxxxx" }
scenario:
turns:
- say: "Hi, I need to change the email on my account."
expect: { contains_any: ["verify", "confirm", "account number"] }
expect_reply_within_ms: 2500
- say: "It's 4 4 8 1 2 0."
expect: { intent: "asks_for_new_email", judge: rubric }
- say: "Actually, never mind."
interrupt_after_ms: 800
expect_agent_stops_within_ms: 600
end_state: { any_of: [hangup_by_agent, transfer_offered] }
thresholds:
time_to_first_audio_ms: { warn: 1500, fail: 3000 }
turn_latency_p95_ms: { warn: 2000, fail: 4000 }
scenario_result: { expect: pass }
The same file format describes a WebRTC room or a TURN check, so a voice agent and the infrastructure under it live in one configuration. The full reference is in the documentation.
Before release and after
Pre-release simulation runs hundreds of synthetic callers with varied personas and accents against a build and reports a pass rate. A canary runs a handful of critical flows every few minutes against production and reports the first run that fails. The two answer different questions: "is this build good enough to ship" and "is the shipped agent still good right now". RTCanary is the second one. A flow that already exists as a test case in CI can be reused as a canary scenario, so the set of flows you watch in production is the set you already decided matters.
Targets
Any agent reachable by a phone number, a WebRTC room or a realtime session. In practice that covers agents built on LiveKit Agents, Pipecat, Vapi, Retell, Bland, the OpenAI Realtime API, Gemini Live, ElevenLabs Conversational AI, and in-house pipelines behind Twilio, Telnyx or a SIP trunk. The canary identifies itself with a caller ID and a metadata header you choose, so its calls can be excluded from your analytics and billed to a test account.
Alerts your on-call agent can act on
An alert is a structured document, not a sentence. It carries the run, the failing turn, the attribution and the links to the trace and the recording when recording is enabled for the check.
{
"check_id": "support-change-email",
"region": "us-east",
"result": "fail",
"failed_turn": 2,
"attribution": {
"layer": "agent_pipeline",
"stage": "time_to_first_token",
"evidence": "ttft p95 2.9 s (baseline 0.8 s); media rtt 41 ms, loss 0 %"
},
"scenario": { "expected": "asks_for_new_email", "judge": "rubric", "verdict": "fail", "reason": "agent asked for the account number a second time" },
"trace_url": "https://app.rtcanary.com/runs/…"
}
Through the MCP server, Claude Code, Cursor or your own agent can list checks, fetch this run, read the per-second trace, compare it with the last passing run and mute the check for the duration of a rollback. The same calls are available as a REST API; the reference is in the documentation.
Questions
Does the canary need access to the agent's internals?
No. It measures from the outside, as a caller. If your framework posts per-turn stage metrics (end of utterance, time to first token, TTS start) to the run's webhook, the attribution splits agent time from network time; without them it still reports time to first audio and the media leg.
How is "the agent answered correctly" judged?
Deterministic checks first: contains, regex, a DTMF tone, a transfer event, a hang-up. A rubric judge is enabled per turn when you need an intent check; you write the rubric, the judge's verdict and reasoning are stored with the run, and a judge failure is reported separately from a latency failure so the two are never confused.
Will canary calls pollute my analytics?
The canary uses a caller ID and a metadata header you configure. Exclude them in your analytics and route them to a test account on your agent platform; most platforms support both by caller ID.
What does it cost?
A scenario is one check per region, the same as a room or TURN check. The phone leg is included. See pricing.