Synthetic monitoring for voice and video: why a 200 OK is not a working call

Synthetic monitoring is a scheduled, scripted request sent from a known location to confirm that a system still works, without waiting for a real user to hit the problem. Datadog describes its synthetic tests as "simulated requests and actions from around the globe"; Grafana calls it black-box monitoring: checks that "continuously run against remote targets". For a voice or video service the catch is that none of the checks the mainstream platforms ship places a call. A call can return 200 OK on signaling and still carry no audio, because the media path fails separately: ICE never finds a working candidate pair, the TURN server refuses an allocation, the SDP advertises a private address, or the two ends agree on no codec. Catching that needs a check that places a call, not a request.

What is synthetic monitoring?

Synthetic monitoring runs the same test on a timer from one or more probe locations and records whether it passed and how long it took. The Google SRE book files this under black-box monitoring: "testing externally visible behavior as a user would see it". The four large platforms define it by the kinds of test they can run:

• Datadog: API tests (HTTP, SSL, DNS, WebSocket, TCP, UDP, ICMP, gRPC), browser tests, mobile application tests, and private locations for internal targets.
• Grafana Cloud: Ping, HTTP, DNS, TCP, Traceroute, MultiHTTP, k6 scripted and k6 browser checks, run from public probes or private probes you install yourself.
• New Relic: Ping, simple browser, scripted browser, step monitor, API test, broken links and certificate check. Its docs say a ping monitor "simply checks to see if an application is online".
• Dynatrace: single-URL browser monitors, browser clickpaths, HTTP monitors and network availability monitors (ICMP, TCP, DNS), the last of which run only on private synthetic locations.

Three things synthetic monitoring is not. It is not real user monitoring: RUM instruments the sessions of actual users (Datadog's definition is "end-to-end visibility into the real-time activity and experience of individual users") and sees nothing when nobody is using the product. It is not load testing: a synthetic check simulates one user per execution, not a crowd. And it is not an uptime ping: a ping proves a host answers ICMP, which says nothing about whether the service on it does its job.

RUM vs synthetic monitoring: which one tells you a call is broken?

Both, but at different moments. RUM reports a broken call after a user has suffered it; a synthetic check can report it at 03:00 when the room is empty. The two also measure different populations: RUM covers every device, network and browser your users actually have, while a synthetic probe covers the few locations and configurations you chose.

For real-time media the split matters more than for a web page. A page that loads slowly is still a page; a call with no return audio is not a call. RUM for WebRTC means collecting getStats() from users' browsers, the same counters a canary reads from its own, so the useful combination is a canary for detection and RUM for population-wide confirmation, not one or the other.

Why does an HTTP synthetic pass while the call is dead?

Because signaling and media are different connections with different failure modes. The join endpoint, the token service and the room API answer over HTTPS or WebSocket and are exactly what an HTTP synthetic exercises. Audio and video travel over a UDP path that is negotiated after signaling succeeds, and that negotiation can fail while every HTTP assertion stays green. Four failures account for most "dead calls" with a healthy dashboard.

ICE finds no working pair

Interactive Connectivity Establishment (RFC 8445) tries candidate pairs until one passes a connectivity check. The checklist reaches the Failed state when there is no valid pair for each component and "all of the candidate pairs in the checklist are in either the Failed or the Succeeded state". The browser surfaces this as the failed ICE transport state, which the WebRTC specification calls "a terminal state until ICE is restarted". The disconnected state is different: the spec describes it as "a transient state that may trigger intermittently (and resolve itself without action) on a flaky network". A check that treats both the same will page you for Wi-Fi hiccups and miss the real outage.

TURN refuses the allocation

When host and reflexive candidates fail, media is relayed through TURN (RFC 8656). RFC 8445 recommends a type preference of 0 for relayed candidates, which it defines as "only be used as a last resort", so a relay is the path taken when direct connectivity fails, which is where restrictive networks put your users. A TURN server can be up on its port and still refuse to relay: RFC 8656 lists 401 (bad credentials), 437 Allocation Mismatch, 486 Allocation Quota Reached and 508 Insufficient Capacity ("no more relayed transport addresses available"). An allocation also expires: "the default lifetime of an allocation is 10 minutes", and the client must send a Refresh before it lapses. A TCP check on the TURN port sees none of this.

One-way audio from a private address

SIP and WebRTC both carry the media address inside the session description. RFC 6314 describes the classic failure: a client behind a NAT advertises a private address to receive media, "and in the case of media such as RTP, this will result in no audio traversing NATs". Signaling completed, the call is "up", one side hears nothing. RFC 6314 also warns that SIP-aware application layer gateways in NATs "can have an adverse impact on the functionality of SIP".

No common codec

If the offer and answer share no codec the SIP side returns 488 Not Acceptable Here (RFC 3261 §21.4.26). A synthetic that only checks the room API cannot know, because the API never saw the SDP.

What does each kind of check catch?

The table maps the four check types against the failures above. "Uptime check" means ICMP/TCP; "HTTP synthetic" means an API or browser test against the signaling and web tier; "call canary" means a probe that joins a real room or dials a real trunk and inspects media statistics.

How does a call canary work?

A call canary is a synthetic check whose "request" is a call. It has three layers, each of which can be run on its own.

1. STUN/TURN probe. A STUN Binding request (RFC 8489) returns the probe's reflexive address in the XOR-MAPPED-ADDRESS attribute; a TURN Allocate request (RFC 8656) proves the relay will hand out a relayed address with the credentials your clients use, and the response code tells you why not. Coturn ships turnutils_stunclient ("a basic STUN client") and turnutils_uclient (a test client "supplied for the test purposes only") that do exactly this from the command line.
2. WebRTC room canary. A headless Chromium joins the room through your join URL or token. Chromium's testing flags make this deterministic: --use-fake-device-for-media-stream "feeds a test pattern to getUserMedia() instead of live camera input", --use-fake-ui-for-media-stream "avoids the need to grant camera/microphone permissions", and --use-file-for-fake-video-capture plays a Y4M file. The probe then polls RTCPeerConnection.getStats() and reads the W3C statistics objects described below.
3. SIP canary. For a trunk or PBX, a SIP OPTIONS request (RFC 3261 §11) queries a server "as to its capabilities" without ringing anyone and confirms reachability; an INVITE with reference audio exercises the media path, with the receiving side's RTCP reports supplying loss, jitter and round-trip time.

The scheduling, regions and alert routing around these three layers are what RTCanary provides; the how it works page lists what each check records.

Which metrics should a voice or video canary record?

Record the state transitions first and the quality numbers second: a canary that logs "ICE failed at 00:04.2 after the relay candidate was tried" is worth more than a MOS estimate with no context. The W3C WebRTC Statistics API defines every field below.

The RFC 3550 jitter estimator is a running average: J(i) = J(i−1) + (|D(i−1,i)| − J(i−1))/16, where D is the difference in transit time between consecutive packets. A single late packet moves it by a sixteenth of the deviation, so a spike on a graph means sustained variation, not one outlier. For delay, ITU-T G.114 (2003) is the planning reference: below 150 ms mouth-to-ear, "most applications, both speech and non-speech, will experience essentially transparent interactivity"; above 400 ms is "unacceptable for general network planning purposes".

How do you set thresholds without alert fatigue?

Alert on state, not on every number. The SRE book's rule is that "every page should be actionable"; a page that says "jitter rose in Frankfurt" is not, while "ICE failed from 3 of 3 probes in Frankfurt for 5 minutes" is. Four practices follow from the specifications and the vendor docs above:

• Require confirmation across runs and locations. Datadog's alerting rule is a template: "an alert is triggered if any assertion fails for X minutes from any n of N locations", with retries "X times after Y ms" before a result counts as failed. One probe failing once is a data point; the same failure from two regions on consecutive runs is an incident.
• Separate failed from disconnected. The spec itself says the latter may resolve without action. Page on failed; graph disconnected.
• Treat a change of candidate type as a warning, not a page. A probe that connected direct yesterday and over TURN today is telling you something changed in the network, but the call works.
• Use absolute bounds only where a standard gives one. G.114's 150 ms and 400 ms delay bands and the R-factor scale in RFC 3611 are documented; a "good" jitter number is not, so set jitter and loss thresholds from your own probe's baseline and alert on departure from it.

What synthetic monitoring does not catch

A canary measures the path from the probe to your stack. It does not measure your users.

• The user's device and network. Microphone permission denied, a muted hardware switch, a congested home Wi-Fi or a corporate firewall that blocks UDP are invisible to a probe sitting in a data center. Only RUM sees them.
• Regions without a probe. An ICE failure caused by a single carrier in one country shows up only if a probe sits behind that carrier. Public probe locations sit in cloud data centers, not on access networks.
• Private networks. A stack reachable only from inside the office needs a private probe; Datadog, Grafana and Dynatrace all ship one for this reason, and Dynatrace's network availability monitors run nowhere else.
• Long-call degradation. A 30-second canary will not see a jitter buffer that collapses after twenty minutes, or a TURN allocation that was never refreshed after its 10-minute default lifetime. Checks that hold the call longer cost more and find different faults.
• Perceived quality. An E-model estimate is a prediction from network parameters, not a score from listeners (ITU-T P.800.1 is the reference for MOS terminology). The estimate is useful for trend and comparison, not as a claim about what a listener heard.

Questions engineers ask before adding a call canary

Is synthetic monitoring the same as uptime monitoring?

No. An uptime check answers "does the host respond" over ICMP or TCP. A synthetic check executes a scripted transaction and asserts on the result; New Relic's ping monitor is the simplest synthetic type and the two overlap only there. A call canary is a synthetic check whose transaction is a call.

Does RUM make synthetic monitoring unnecessary?

No. RUM needs users to generate data, so it cannot detect a fault before the first affected call or in an idle region, and it cannot test a release before rollout. Synthetic cannot see user devices. Each covers the other's blind spot.

Can a call canary run inside a private network?

Yes, with a private probe: the same model Datadog private locations and Grafana private probes use for internal HTTP targets. The probe needs outbound reach to your signaling, STUN/TURN and media hosts and nothing inbound.

Sources

1. Datadog Docs: Synthetic Monitoring and HTTP tests: alert conditions; Real User Monitoring
2. Grafana Cloud Docs: Synthetic Monitoring, check types, private probes
3. New Relic Docs: Types of synthetic monitors
4. Dynatrace Docs: Synthetic Monitoring and Network availability monitoring
5. Google SRE Book, ch. 6: Monitoring Distributed Systems
6. RFC 8445: Interactive Connectivity Establishment (ICE)
7. RFC 8656: Traversal Using Relays around NAT (TURN)
8. RFC 8489: Session Traversal Utilities for NAT (STUN)
9. RFC 3550: RTP, §6.4.1 receiver report fields
10. RFC 3611: RTCP Extended Reports, §4.7 VoIP Metrics
11. RFC 3261: SIP, §11 OPTIONS and §21.4.26 488 Not Acceptable Here
12. RFC 6314: NAT Traversal Practices for Client-Server SIP
13. W3C: Identifiers for WebRTC's Statistics API; W3C WebRTC 1.0, RTCIceTransportState
14. ITU-T G.107 (06/2015): The E-model; ITU-T G.114 (05/2003): One-way transmission time; ITU-T P.800.1 (07/2016): MOS terminology
15. WebRTC project: Testing (Chromium fake media flags)
16. coturn: README.turnutils