Voice agent latency: how to measure it end to end

Voice agent latency is the time from the moment a caller stops speaking to the moment the agent's first audio reaches the caller. It is a sum of stages: the endpointing wait, the final transcript, the LLM's time to first token, the TTS time to first byte, and two network legs with a jitter buffer at the end. LiveKit's documentation reduces the pipeline part to end_of_utterance_delay + llm.ttft + tts.ttfb; Pipecat measures it as the gap from the VAD's user-stopped-speaking frame to the bot-started-speaking frame. Neither number includes the media network legs, and no vendor reviewed here publishes an end-to-end figure in its docs. To know what the caller hears, measure time to first audio from outside the pipeline and reconcile it with the pipeline's own timestamps.

What goes into the latency budget of a voice agent

Each stage is measured by a different party, which is why pipeline metrics and caller experience rarely agree. The table lists the stages in order, who can time them, and what the vendor docs say about the defaults that set the floor.

Two of the stages are decisions, not transfers. Endpointing is a wait the framework chooses: Pipecat's Smart Turn and LiveKit's turn detector both run on top of VAD and lengthen or shorten the wait by content, so the same pipeline has a different endpointing delay for "What's my balance?" than for "I was wondering whether…". The second decision sits on the caller's side: the jitter buffer trades delay for smooth playout and grows under jitter without any change in the pipeline.

What vendors state about latency in their documentation

The list below is limited to what the documentation says, not blog posts or launch announcements. Where the docs give no number, that is recorded as such.

• OpenAI Realtime API. The guide positions the API for "low first-audio latency" and recommends WebRTC in the browser "rather than WebSockets for more consistent performance"; the WebRTC page adds that WARP "reduces the time it takes to start a Realtime API voice session". Turn detection defaults to server_vad; semantic_vad has an eagerness setting where auto equals medium. No millisecond figures for response latency appear in the Realtime, VAD, WebRTC or SIP guides.
• LiveKit Agents. The metrics reference defines ttft, ttfb, end_of_utterance_delay, transcription_delay and e2e_latency ("from when the user stopped speaking to when the agent began responding") and gives the approximation total_latency = eou.end_of_utterance_delay + llm.ttft + tts.ttfb. The turn detector page states figures only for the deprecated text model: about 50–160 ms per turn on CPU. The knowledge base names TTFT and TTFB as "the most important latency metrics to focus on initially" and recommends placing the agent close in network terms to the LLM, STT and TTS services.
• Pipecat. Metrics are TTFB, TTFA, TTFAT, processing time and text aggregation time, enabled with enable_metrics=True. The UserBotLatencyObserver times VADUserStoppedSpeakingFrame to BotStartedSpeakingFrame, adjusted for the VAD stop_secs; its docs show an example breakdown (0.200 s endpointing, 0.125 s transcription, 0.336 s LLM, 0.359 s synthesis, 1.020 s total) that is illustrative, not a benchmark. Smart Turn inference runs "on low-cost cloud instances in under 100ms".
• Deepgram. endpointing is on by default at 10 ms and sets speech_final; utterance_end_ms accepts 1,000 to 5,000 ms because interim results arrive about once a second. No end-to-end latency figure on those pages.
• ElevenLabs. The latency guide gives the ~75 ms Flash figure, the regional TTFB table above, and the caveat that "Actual end-to-end latency will vary with factors such as your location & endpoint type used". Higher-quality output formats add latency; auto_mode on websockets avoids waiting for a text chunk schedule.
• Vapi. Defines voice-to-voice latency as "the gap between when a caller finishes speaking and when your assistant starts responding", with five stages (transport, endpointing, transcription, model response, speech synthesis). Its displayed total is a sum of medians that excludes endpointing and transport, and the docs say so.
• ITU-T G.114. Not a voice-AI document, but the one network reference people cite: it recommends that "a one-way delay of 400 ms should not be exceeded for general network planning" and adds that highly interactive tasks "can be affected by much lower delays". This is one-way transmission time of a network, not turn latency of an agent; adding the two up as if they were the same quantity is a common error.

Which metrics to record per turn

Record every turn, not averages per call. One slow turn caused by a tool call or an endpointing timeout is invisible in a call average and obvious in a p95 of turns.

1. Time to first audio (TTFA). From the end of the caller's speech to the first non-silent agent audio at the caller. This is the number the caller perceives. Store p50 and p95 per region and per hour.
2. Pipeline turn latency. LiveKit's e2e_latency or the sum of end_of_utterance_delay + ttft + ttfb; Pipecat's on_latency_measured value. Store next to TTFA with the same turn id so the two can be subtracted.
3. Stage split. Endpointing wait, transcription delay, TTFT, TTS TTFB, each per turn. The stage that moves when p95 moves is the one to fix.
4. Barge-in response. From the caller starting to speak while the agent is talking to the last agent audio packet received. OpenAI's interrupt_response and the frameworks' interruption handling decide whether the agent stops; the number tells you how fast.
5. First response after connect. Pipecat exposes it as on_first_bot_speech_latency, from ClientConnectedFrame to the first BotStartedSpeakingFrame. It includes session setup and is usually the longest "turn" of the call.
6. Media leg. roundTripTime from remote-inbound-rtp, jitter and packetsLost from inbound-rtp, jitterBufferDelay / jitterBufferEmittedCount, sampled every few seconds. Loss and jitter do not change TTFA much but they change what the caller hears, which is the other half of the picture; MOS is the metric for that half.

How to tell network delay from model delay

The pipeline timestamps and the client statistics answer different questions, and the difference between them is the diagnosis. getStats() gives the network part: candidate-pair.currentRoundTripTime is "the latest round trip time measured in seconds, computed from both STUN connectivity checks", and remote-inbound-rtp.roundTripTime is the estimate "based on the RTCP timestamps in the RTCP Receiver Report". The pipeline gives the model part: the framework metrics listed above, stamped on the agent's clock.

// Sample the media leg next to each turn (browser side)
async function mediaLeg(pc) {
  const out = { rttStun: null, rttRtcp: null, jitter: null, lost: null, bufferMs: null };
  for (const s of (await pc.getStats()).values()) {
    if (s.type === "candidate-pair" && s.nominated) out.rttStun = s.currentRoundTripTime;
    if (s.type === "remote-inbound-rtp" && s.kind === "audio") out.rttRtcp = s.roundTripTime;
    if (s.type === "inbound-rtp" && s.kind === "audio") {
      out.jitter = s.jitter;
      out.lost = s.packetsLost;
      if (s.jitterBufferEmittedCount) {
        out.bufferMs = 1000 * s.jitterBufferDelay / s.jitterBufferEmittedCount;
      }
    }
  }
  return out;
}

With both sides stored per turn, the split is arithmetic:

unexplained = TTFA_at_caller - pipeline_turn_latency - RTT
            ≈ jitter buffer + encode/decode + anything the pipeline does not stamp

If unexplained grows while the pipeline metrics stay flat, the cause is on the media leg or in the client's buffer. If TTFT or TTFB grow with a flat unexplained, the cause is the model vendor, the region, or the prompt size, and the network is innocent. If the endpointing stage grows by itself, the turn detector is waiting on content: a semantic detector lengthening the wait for trailing-off speech, or a max_delay timeout firing on calls with background noise.

Telephony adds a leg no framework stamps. A phone call that reaches OpenAI through a SIP trunk (sip:$PROJECT_ID@sip.api.openai.com;transport=tls) carries the carrier's delay before any pipeline timestamp exists, and there is no getStats() on the PSTN leg. The only way to measure that path is to place a call into it from a device that records both the prompt and the response.

How to set a synthetic canary on a voice agent

A canary turns the measurement above into a scheduled test with a fixed script, so a change in TTFA is a change in the system, not in what the caller said.

1. Fix the prompt. One recorded utterance, same file every run, ending in a clear sentence boundary. Keep a second utterance that trails off, to measure the semantic endpointing path separately.
2. Join as a real client. A headless browser over WebRTC, or a SIP call for a phone-facing agent. The canary sends real encoded audio through the real ICE/TURN path or trunk, so the network legs are included.
3. Mark the end of the prompt on the canary's clock at the last non-silent sample sent.
4. Detect first agent audio at the canary: the first received frame above an energy threshold, after decoding. TTFA is the difference. Record the media-leg statistics in the same second.
5. Test barge-in on the next turn: start the second utterance while the agent is still speaking and time the gap to the last agent frame received.
6. Join the two records. Tag the canary call with an id the pipeline logs, pull the framework metrics for the same turn, and store TTFA, pipeline latency and unexplained side by side.
7. Run from more than one region and alert on the p95 of the last N runs per region, not on a single run.

# canary pseudocode
t_prompt_end = send_audio(prompt_wav)             # last non-silent frame on the wire
t_first_audio = await first_frame_above(threshold) # first decoded agent frame
ttfa = t_first_audio - t_prompt_end
stats = media_leg(pc)
pipeline = fetch_turn_metrics(call_id)             # eou, ttft, ttfb from the framework
unexplained = ttfa - (pipeline.eou + pipeline.ttft + pipeline.ttfb) - stats.rttStun
record(region, ttfa, pipeline, stats, unexplained)

RTCanary's voice-agent check runs this loop on a schedule: it dials the agent over WebRTC or phone, plays the scripted prompt, and reports time to first audio, turn latency and barge-in handling per region, see monitoring for voice agents. The method is the same whether the canary is a product or a script in a container.

Limitations

• An LLM is not deterministic. The same prompt yields different completions; TTFT is more stable than full duration, and a tool call inside the turn adds a variable wait no stage metric separates out unless the framework stamps it.
• Energy-based first-audio detection counts a filler ("Let me check that") as first audio. Measuring time to first substantive audio requires transcribing the agent's reply in the canary, which adds the canary's own STT latency and errors.
• Vendor figures are measured at the vendor's edge, for one model, one region and one output format. ElevenLabs says so explicitly; the others state nothing, which is not a guarantee.
• A canary runs from its own regions and networks. It does not see a caller on a congested mobile link or behind a corporate proxy; its p95 is the system's, not the population's.
• On the PSTN leg there are no WebRTC statistics; the canary can time the call but cannot split carrier delay from the trunk's.
• Semantic endpointing makes latency content-dependent by design. One prompt measures one point of that curve; a trailing-off prompt measures another, and neither is "the" latency.
• The arithmetic split assumes the canary's and the agent's clocks agree; without NTP on both, unexplained absorbs the skew.

FAQ

What is a good voice agent latency?

None of the vendor docs reviewed here states a target in milliseconds, and ITU-T G.114's 400 ms is a one-way network planning limit, not a turn-latency threshold. Set the budget from your own measurements: the p95 of TTFA your callers currently get, split by stage, and a decision about which stage has room. Numbers quoted in blog posts rarely say what was measured between which two events.

Is time to first audio the same as TTS time to first byte?

No. TTS TTFB is one stage, timed on the agent's side from the request to the TTS service to the first audio byte back. Time to first audio is timed at the caller and includes endpointing, STT, the LLM's first token, TTS TTFB, the network and the jitter buffer.

Does WebRTC or WebSocket transport change latency?

OpenAI recommends WebRTC for browser clients "for more consistent performance" and offers WebSocket for trusted servers; the docs publish no comparative numbers. The difference the transport makes is on the media legs, so it shows up in TTFA and in the unexplained term, not in the pipeline's own metrics.

Sources

1. LiveKit Agents, data hooks and metrics: metric definitions, e2e_latency, the total_latency approximation.
2. LiveKit Agents, turn detector: min_delay / max_delay defaults, text model figures.
3. LiveKit knowledge base, reducing latency in voice pipeline agents: TTFT and TTFB, network proximity.
4. Pipecat, metrics: TTFB, TTFA, TTFAT, enabling metrics.
5. Pipecat, UserBotLatencyObserver: measurement frames and example breakdown.
6. Pipecat, STT latency tuning: TTFS definition and built-in P99 values.
7. Pipecat, Smart Turn and Silero VAD analyzer: stop_secs, inference time.
8. OpenAI, Realtime API, voice activity detection, WebRTC, SIP: transports, VAD parameters, SIP endpoint.
9. Deepgram, endpointing and UtteranceEnd: defaults and ranges.
10. ElevenLabs, latency optimization: Flash inference figure, regional TTFB table.
11. Vapi, understanding latency: voice-to-voice definition and what the displayed total excludes.
12. W3C, Identifiers for WebRTC's Statistics API: currentRoundTripTime, roundTripTime, jitter, packetsLost, jitterBufferDelay.
13. ITU-T G.114, One-way transmission time: 400 ms planning limit.