TURN server monitoring: what to check on coturn and managed TURN

TURN server monitoring means proving, on a schedule and from outside your network, that a client can complete a TURN Allocate with valid time-limited credentials, receive a relayed candidate, and push packets through the relay and back on every transport you advertise (UDP and TCP on 3478, TLS on 5349 or 443). A process check or a ping does not prove that: coturn can be up and still fail every allocation because the shared secret rotated, the certificate behind turns: expired, or the allocation quota is full. The checks below use the STUN and TURN messages defined in RFC 8489 and RFC 8656, coturn's own client tools and Prometheus endpoint, and a browser RTCPeerConnection forced to relay-only.

What breaks on a TURN server and what each failure looks like

Most TURN outages are not "the server is down". The daemon answers STUN Binding requests, the port is open, and allocations still fail. The table maps each failure to the symptom a client sees and the signal that catches it.

The last row is why a check that allocates once and exits misses a class of failures: only a probe that holds the allocation across a refresh cycle exercises lifetime handling.

Which checks catch those failures

Four checks cover the table above. Run them from a host outside the TURN server's own network, because a probe in the same VPC reaches the relay over host candidates and never exercises the NAT, the firewall, or the public address mapping.

STUN Binding request

A Binding request (method 0x0001) returns the client's reflexive address in XOR-MAPPED-ADDRESS (RFC 8489 §6.3.1.1). It proves the listener is reachable on 3478 and measures round-trip time:

turnutils_stunclient -p 3478 -c -i 1000 -t 3000 turn.example.com
# -c  continuous binding requests, reports round-trip latency
# -i  interval between requests, ms (default 1000)
# -t  response timeout, ms (default 3000)

A passing Binding check does not mean TURN works. Binding needs no credentials and no relay port, so it passes while the secret is wrong and the relay range is firewalled.

TURN Allocate and relay round-trip with time-limited credentials

With --use-auth-secret, coturn expects the username timestamp:userid and the password base64(hmac-sha1(shared-secret, username)), the scheme of the TURN REST API draft (draft-uberti-behave-turn-rest-00 §2.2, which recommends a TTL of one day, 86400 seconds). The probe must generate a fresh pair every run; a credential baked into a config file is itself a timer. turnutils_uclient does the HMAC when given the secret:

# UDP: allocate, then 5 messages through the relay and back (-y = client-to-client via relay)
turnutils_uclient -v -y -W "$TURN_SECRET" -n 5 -e 203.0.113.10 -p 3478 turn.example.com

# TCP to the server
turnutils_uclient -v -y -t -W "$TURN_SECRET" -n 5 -e 203.0.113.10 -p 3478 turn.example.com

# TLS to the server on 5349
turnutils_uclient -v -y -S -W "$TURN_SECRET" -n 5 -e 203.0.113.10 -p 5349 turn.example.com

-e is the peer address; with -y the tool opens a second client and routes packets between the two through the relay, so a completed run covers Allocate, CreatePermission and the round-trip. For managed TURN (Twilio NTS, Cloudflare, Xirsys) the secret stays with the vendor: fetch credentials from their API first and pass them as -u and -w. Twilio's tokens last 24 hours unless you lower Ttl; Cloudflare's ttl is set per request to generate-ice-servers; Xirsys dynamic credentials default to a 60-second lifetime, short enough that a probe which fetches credentials and then waits fails for a reason unrelated to the relay.

Certificate expiry on the TLS listener

echo | openssl s_client -connect turn.example.com:5349 -servername turn.example.com 2>/dev/null \
  | openssl x509 -noout -enddate

Alert when the remaining validity is shorter than the time it takes you to renew by hand, not at zero days. coturn reads the files given in --cert and --pkey (defaults turn_server_cert.pem, turn_server_pkey.pem), so a renewed certificate on disk is half the job; the check must hit the live listener.

Prometheus scrape

Internal counters, covered next. They tell you what the server saw, never about clients that could not reach it.

Which coturn Prometheus metrics are worth an alert

Start coturn with --prometheus. It listens on port 9641 (--prometheus-port), serves metrics under /metrics, and answers GET / with 200 and the body ok as a health check. The option is unavailable in the apt package; you need a build with libmicrohttpd. The names below are the ones registered in src/apps/relay/prom_server.c.

Two caveats from the source. Traffic counters represent finished sessions, so a dashboard built on them lags every call by the call's duration. And --prometheus-username-labels, which adds a user label, is off by default because every REST-API call creates a new label value; coturn's option text warns this "may cause memory leaks".

# Credentials rejected as expired in the last 5 minutes
rate(turn_auth_credential_failures{cause="expired"}[5m]) > 0

# Allocations near the configured global quota (TOTAL_QUOTA = your --total-quota)
turn_total_allocations > 0.9 * TOTAL_QUOTA

# Exporter or server gone
absent(turn_total_allocations)

The exporter has no allocation-latency histogram, no success ratio, and no per-transport breakdown of Allocate outcomes. Those come from the external probe.

How to test TURN from a browser with relay-only ICE

Setting iceTransportPolicy: "relay" on RTCPeerConnection makes the browser consider only candidates "whose IP addresses are being relayed, such as those being passed through a TURN server" (MDN). A gathered candidate of type relay is therefore a successful Allocate through the browser's own ICE stack, with its own transport selection and retries. The snippet gathers candidates and reports the relay protocol and server URL from getStats():

async function turnProbe(iceServer, timeoutMs = 8000) {
  const pc = new RTCPeerConnection({
    iceServers: [iceServer],          // { urls, username, credential } from your credential service
    iceTransportPolicy: "relay"
  });
  pc.createDataChannel("probe");
  const t0 = performance.now();
  const relay = await new Promise(async (resolve) => {
    const timer = setTimeout(() => resolve(null), timeoutMs);
    pc.onicecandidate = (e) => {
      if (e.candidate && e.candidate.type === "relay") {
        clearTimeout(timer);
        resolve({ ms: performance.now() - t0, candidate: e.candidate });
      }
    };
    await pc.setLocalDescription(await pc.createOffer());
  });
  let relayProtocol = null, url = null;
  for (const s of (await pc.getStats()).values()) {
    if (s.type === "local-candidate" && s.candidateType === "relay") {
      relayProtocol = s.relayProtocol;   // udp | tcp | tls
      url = s.url;                       // the TURN URL the candidate came from
    }
  }
  pc.close();
  return relay ? { ok: true, allocateMs: relay.ms, relayProtocol, url } : { ok: false };
}

Run it once per advertised URL, one urls entry at a time, so a UDP failure cannot be masked by a TCP success. The W3C statistics spec defines relayProtocol as "the protocol used by the endpoint to communicate with the TURN server" and url as "the URL of the ICE server from which the candidate was obtained". For a manual check, the Trickle ICE sample page at webrtc.github.io takes a URI, username and password, has an IceTransports switch (all / relay), and lists each candidate with type, protocol, address, port, URL and relayProtocol.

A relay candidate proves the Allocate. To prove the relay forwards media, create two peer connections in the same page, both relay-only, exchange their offers and answers locally, open a data channel and time a message round-trip; candidate-pair.currentRoundTripTime from getStats() adds the STUN connectivity-check RTT of the pair in use. The general method, scheduled test calls through the real stack instead of counters on the server, is described in synthetic monitoring for voice and video; RTCanary's TURN/STUN check runs the Binding, Allocate and relay round-trip from several regions and checks the TLS certificate, see how it works.

Alert thresholds for TURN checks

Few thresholds come from the protocol; the rest come from your own baseline. RFC 8489 §6.2.1 says the initial retransmission timeout "SHOULD be greater than or equal to 500 ms" and the retransmission count Rc "SHOULD have a default of 7", so a browser behind a silent-dropping firewall spends tens of seconds before giving up on one server. A probe timeout of a few seconds is a stricter bar, which is what an alert needs.

When the standard advice fails

Double NAT and address-dependent mapping. RFC 8656 §1 states that when both hosts are behind NATs with address-dependent or address-and-port-dependent mapping, "hole punching techniques generally fail", and the relay is the only path. A probe with iceTransportPolicy: "all" may still connect through a server-reflexive pair from its own friendlier network and report success; the relay-only policy is the only way to make the probe fail the way those users fail.

Corporate firewalls that allow only 443 over TLS. If the ICE servers list has turn:host:3478?transport=udp and turn:host:3478?transport=tcp only, those users get no candidate at all. Managed providers publish a 443 path: Twilio lists turn:global.turn.twilio.com:443?transport=tcp, Cloudflare lists TURN over TLS on 5349 with 443 as the alternate port. On coturn, --tls-listening-port defaults to 5349 and can be pointed at 443, with --alt-tls-listening-port for a second port. Whatever you advertise, the probe set must include a turns: URI on that port, because a working UDP path says nothing about the TLS one.

TURN server itself behind NAT (cloud instances with a private interface). The relayed address in XOR-RELAYED-ADDRESS must be reachable by the peer. coturn's --external-ip maps the private relay address to the public one, under a condition its option text spells out: "NAT must forward ports directly (relayed port 12345 must be always mapped to the same 'external' port 12345)". With the wrong mapping, Allocate succeeds, the relay candidate looks fine, and nothing flows, which is the case the relay round-trip check exists for.

The probe runs next to the server. A check from the same subnet uses host candidates and the private interface, so it passes while the public path is broken. Put at least one probe location outside the provider's network, more if your users are spread across regions.

Limitations

• A synthetic probe proves the path from the probe's network at the probe's time. It does not see a user's ISP, corporate proxy, or captive portal.
• coturn's traffic counters cover finished sessions, so they describe the past, not the current call.
• turnutils_uclient exercises coturn's client implementation, not a browser's ICE agent; when the two disagree, the browser is the one users run.
• Managed TURN exposes no Prometheus endpoint; everything you know comes from the external probe and the vendor's console.
• The relay-only browser test needs a page or a headless browser; it cannot run from a plain cron job.
• Credential TTLs differ per vendor (Xirsys 60 seconds, Twilio 24 hours by default), so a probe must fetch credentials immediately before each run or it measures its own staleness.

FAQ

Does a successful STUN Binding mean the TURN server works?

No. Binding needs no credentials and no relay port. It passes while the shared secret is wrong, while the relay port range is firewalled, and while the TLS certificate is expired, because it runs over UDP 3478 without authentication.

How often should a TURN probe run?

Often enough to fit inside your credential TTL, and long enough to cover lifetime handling: the default allocation lifetime in RFC 8656 is 10 minutes, so a probe that holds an allocation across one refresh runs at least that long or requests a shorter LIFETIME and refreshes it. The interval between runs has no protocol-given value; it trades alert delay against the allocations you spend.

Can Twilio NTS, Cloudflare or Xirsys TURN be monitored the same way?

Yes, with the external checks: fetch credentials from the vendor API, run the Allocate and relay round-trip against each advertised URI and transport, and check the certificate on the turns: endpoint. The Prometheus section does not apply; the vendor runs the server.

Sources

1. RFC 8656, Traversal Using Relays around NAT (TURN): allocation lifetime, permissions, channel bindings, error codes, default ports.
2. RFC 8489, Session Traversal Utilities for NAT (STUN): Binding method, XOR-MAPPED-ADDRESS, RTO and Rc defaults, 401 and 438.
3. RFC 7065, TURN URIs: turn: / turns: schemes and the ?transport= parameter.
4. RFC 8445, ICE: candidate types, connectivity checks as STUN Binding requests.
5. draft-uberti-behave-turn-rest-00, A REST API For Access To TURN Services: time-limited credential format and TTL recommendation (expired individual draft).
6. coturn README.turnserver: option reference (--prometheus, quotas, bandwidth, TLS, REST API, --external-ip).
7. coturn README.turnutils: turnutils_uclient and turnutils_stunclient options.
8. coturn prom_server.c: Prometheus metric names, labels and help text.
9. MDN, RTCPeerConnection() constructor: iceTransportPolicy and iceServers.
10. W3C, Identifiers for WebRTC's Statistics API: relayProtocol, url, currentRoundTripTime.
11. WebRTC samples, Trickle ICE: manual relay candidate test.
12. Twilio, Network Traversal Service: token lifetime and URI list.
13. Cloudflare Realtime TURN and generate credentials: ports, per-allocation limits, ttl.
14. Xirsys docs: dynamic credential lifetime.