STUN and TURN servers: is the relay working?

The STUN / TURN sensor checks the server that VPN meshes, WebRTC video, softphones and other peer-to-peer software lean on to get through NAT. Most of the time such a server is invisible: two devices find each other, connect directly and nobody notices it was involved. When it fails, the symptoms turn up somewhere else entirely. Calls connect but carry no audio, video meetings work from the office but not from home, and a VPN mesh still says "Connected" while some of its peers can no longer reach each other, or crawl through a relay. Checking the server directly says which part is broken before anyone starts on the clients.

What STUN, TURN and ICE are

Two devices behind different NAT routers cannot simply connect to each other: neither knows the other's public address, and neither router will let an unrequested packet in. Three standards work together to get round this.

• STUN answers one question: "what address do you see me at?" A device sends a binding request to the STUN server, and the answer carries the public address and port its packets arrived from. That is the device's server-reflexive address, the one its NAT router is using for it. STUN does nothing else, and carries no traffic.
• TURN is the fallback for when no direct path can be made, usually because one side is behind a strict firewall or a NAT that changes ports for every destination. The device asks the TURN server for an allocation: an address and port on the server itself, the relayed address. Anything sent to the relayed address is passed to the device, and the device can send through it in the other direction. Every packet of the conversation then goes through the server, so TURN costs the operator bandwidth, and that is why it is protected by a password where STUN is not.
• ICE is the procedure that uses both. Each device gathers its candidates: its own local addresses, its server-reflexive address from STUN and its relayed address from TURN. The two devices swap candidates through the application's own signalling channel, test every pair, and use the best one that works: direct where possible, relayed where not.

A STUN or TURN server is often the same program doing both jobs, most commonly coturn, usually on port 3478, UDP and TCP. TURN also needs a range of UDP ports for the relayed addresses (coturn's default is 49152 to 65535). Both must be reachable from the internet, and that includes any port forwarding on a firewall in front.

Authentication

STUN binding requests normally need no authentication at all. A public STUN server answers anyone, because all it gives away is the asker's own address.

TURN is different. An allocation hands out the server's bandwidth, so it always needs a credential. The standard method is the long-term credential mechanism:

1. The client asks for an allocation with no credential.
2. The server refuses it with error 401, and includes a realm (a name for the server's set of users) and a nonce (a value that changes from time to time).
3. The client asks again, this time with its username, the realm, the nonce and a message integrity code. That code is a hash over the whole request, keyed on the username, realm and password together, so the password itself is never sent.
4. The server checks the code. Error 401 again means a wrong username or password. Error 438 means the nonce has gone stale; the client takes the fresh one from the error and tries once more.

Where servers differ is in how the username and password come into being. There are two kinds.

Fixed credentials

The server has a username and password written into its configuration, or a list of them, and they stay the same until somebody changes them. In coturn this is lt-cred-mech with one or more user=name:password lines in turnserver.conf. A self-hosted NetBird whose management.json has TimeBasedCredentials set to false works this way, with the username and password under TURNConfig, Turns.

Fixed credentials are simple, but every client is handed the same password, so anyone who has had a client can use the relay until the password is changed. They are common on private servers that serve one product.

Time-limited credentials

The server and the application share a secret, and the password is worked out from it rather than stored. When a client needs to use the relay, the application's own server makes a credential for it on the spot:

• the username is the time the credential expires, as a Unix timestamp, often followed by a colon and the user's name, for example 1793253600:alice;
• the password is an HMAC-SHA1 of that username, keyed on the shared secret, written in Base64.

The TURN server repeats the same calculation and checks that the time has not passed, so nothing about individual users is stored on it, and a leaked credential stops working by itself within hours. This is often called the TURN REST API method. In coturn it is use-auth-secret with static-auth-secret. It is the usual arrangement for WebRTC conferencing (Jitsi, Nextcloud Talk, BigBlueButton), for Matrix homeservers (turn_shared_secret), and for NetBird with TimeBasedCredentials set to true.

Such a server cannot be given a credential to keep, since any credential would expire within a day. The sensor can instead hold the shared secret and make a fresh credential from it at every check, exactly as the application's own server does, valid for an hour. The secret is sealed where it is stored and reaches the probes with their configuration, held in memory only, as every other credential is. It is still the key that mints credentials for anyone, so hold it on the sensor itself rather than on a site or group, and change it on the server if a probe is ever lost. Without the secret, check these servers with no credential at all: that proves the server is up and answering STUN, though not that it relays.

Which kind does a server use?

• A coturn configuration with use-auth-secret or static-auth-secret means time-limited credentials. lt-cred-mech with user= lines means fixed ones.
• A credential a client has been given that has a long number at the front of the username, and a password that looks like Base64 ending in =, is time-limited.
• Some servers use a third method, OAuth access tokens (RFC 7635). It is rare, and the sensor cannot take part in it: check such a server with no credential, STUN only.

ICE has passwords of its own

The ICE connectivity checks two peers run against each other carry a short username and password too (ice-ufrag and ice-pwd). Each side makes them up for every session and passes them to the other through signalling. They have nothing to do with the TURN server's credential, and nothing to do with checking the server.

How the sensor checks

• Without a credential it sends one STUN binding request and reads the answer: the server is up, answering on that port, and saw the probe at the reported address.
• With a credential it carries on and proves the relay works end to end, which is what a client behind NAT actually depends on. It allocates a relayed address with the credential, permits its own address to send to it, sends a packet to the relayed address from a second socket, and waits for the server to hand it back. That packet comes in from outside through the relay port range, so a range that a firewall does not forward fails here even though the allocation succeeded. Afterwards the allocation is released, so the server is not left holding it.

Everything is over UDP. TURN over TCP and over TLS on port 5349 are not tested.

Setting it up

Create a device for the server with its hostname, add a STUN / TURN server sensor, and choose probes for it.

The sensor is 2 credits a reading. Each check makes one short-lived allocation and sends a few small packets, so it is no burden on the server.

What it reads

The response time is the whole check.

Rules worth having

• The failure itself needs no rule: a server that does not answer, refuses the credential or does not relay gives no reading, and the failure threshold turns that into CRIT as for any sensor.
• SD03 not equal to TURN gives WARN on a sensor that has a credential. It catches the credential going missing, which would otherwise quietly downgrade the check to STUN.
• SV01 or SV03 well above their usual level gives WARN, if the server is shared and slow answers matter. Chart them for a week first to see what usual is.

When it fails

The message names the stage that failed: binding, allocate, relay binding, permission or relay.