How to monitor particular kit with Oversight: which sensor to use, where on the device to set up access, which user function and extraction rows read the response, and how to write the rules that turn it into an alarm. Search for a vendor, a product or an error message, or ask a question in your own words.
UDP and TCP echo: is this leg of the network sound?
The echo sensors test one leg of a network: a LAN link, a leased line, a 4G or Starlink uplink, a tunnel between two sites. A small program, oversight-echo, runs on a machine at the far end of the leg, and a probe at the near end tests against it. Every packet and every byte the probe sends is numbered and its contents known, so anything lost, reordered, duplicated or damaged on the way is counted, in each direction.
There are two sensors, usually used together:
- UDP echo stream sends a set number of packets, one a millisecond, and gets each one back. It answers whether the leg is sound: loss, jitter, round trip, and whether the loss is scattered (weather, a dirty dish, a failing cable) or one outage (a handover, a cell change).
- TCP echo transfer sends a set size up and then back down. It answers how fast the leg is in each direction and how hard TCP had to work for it: retransmits and stalls.
Everything is layer 3. Nothing uses ICMP, which is not reliably forwarded across WAN links.
Before you start
- Run the echo at the far end. Download the build for the machine and run it. Nothing to install and nothing to configure:
- Linux:
https://oversight.im/echo/oversight-echo-linux-amd64,-linux-arm64or-linux-armv7 - Windows:
https://oversight.im/echo/oversight-echo-windows-amd64.exe - macOS:
https://oversight.im/echo/oversight-echo-darwin-arm64or-darwin-amd64
-portpicks another port. It logs each test it serves. - Linux:
- Open the port. The probe must reach the echo on 29732, TCP and UDP. Windows asks the first time the echo runs; a firewall in between needs a rule.
- The echo has no password. It is meant for private legs and for days at a time, not for the internet. Stop it when the work is done.
- Find a forgotten echo by scanning for port 29732.
- Mind the probe groups. The test is from whichever probe polls the sensor. Use a group whose probes sit at the near end of the leg, ideally one probe, or the figures describe a different path each time the sensor moves.
Setting it up
Create a device for the machine running the echo, with its address, and add the sensors to it.
| Setting | What to put |
|---|---|
| Packets (UDP) | How many to send, up to 10000. 1000, the default, takes a second. Start there and raise it to pin a fault down. |
| Size each way (TCP) | Written as 512KB, 1MB or 1GB, up to 1GB; the default is 1MB. A small size mostly measures TCP getting going, so a larger one reads closer to the leg's real speed, at the cost of the data used. |
| Port | Empty for 29732. |
| Interval | Every 5 minutes is a sensible watch; every minute while hunting a fault. The packets must finish within the interval, so a short interval limits the count. |
Both are 2 credits a reading. Mind the data on a metered link: 1MB each way every minute is about 1.4GB a day in each direction. The UDP sensor uses very little.
What it reads
UDP echo stream
| Slot | Reading |
|---|---|
| SV01, SV02, SV03 | Sent, received and lost, in packets. |
| SV04 | Loss, as a percentage of those sent. |
| SV05, SV06 | Lost on the way out and lost on the way back. The echo counts what reaches it, which splits the loss by direction. |
| SV07, SV08, SV09 | Out of order, duplicated and corrupt (arrived, but not as sent). |
| SV10, SV11, SV12 | Round trip minimum, average and maximum, in milliseconds. |
| SV13 | Jitter: how much each packet's round trip differs from the one before, averaged, in milliseconds. |
| SV14 | Longest gap: the longest run of lost packets, in milliseconds (one packet is one millisecond). |
| SV15 | Scattered loss: loss leaving out any single gap over 300 ms, as a percentage. |
TCP echo transfer
| Slot | Reading |
|---|---|
| SV01 | Size each way, in bytes. |
| SV02, SV03 | Upload and download, in Mbps. |
| SV04, SV05 | Upload and download time, in milliseconds. |
| SV06, SV07 | Retransmits up and down, counted by the sending end's own TCP stack. -1 where that end cannot say. |
| SV08, SV09 | Longest stall up and down: the longest wait between pieces arriving, in milliseconds. |
| SV10, SV11 | Corrupt bytes up and down. TCP checks its own data, so anything here got past it and is serious. |
Connect time and response time are filled in too: the average round trip for UDP and the whole run for TCP.
Rules worth having
- SV09 greater than 0 gives CRIT on the UDP sensor, and SV10 or SV11 greater than 0 gives CRIT on the TCP one. Damaged data is never normal.
- SV15 greater than 1 gives WARN, greater than 5 gives CRIT. Scattered loss is the figure rain fade and a dirty dish move, and a phone call starts breaking up at a few per cent. Satellite handovers do not trip it.
- SV14 greater than 2000 gives WARN if outages matter as well: two seconds with nothing through.
- SV06 or SV07 greater than a few dozen gives WARN for a leg that should be clean. A busy link retransmits a little; a sound one should not climb.
An echo that is not running, or a leg that passes nothing, gives no reading at all, and the failure threshold turns that into CRIT as for any sensor.
Pinning a fault down
Run two echoes at once on the same path, one near and one far, each with its own device and sensor from the same probe. When the far one shows loss and the near one does not, in the same minute, the fault is between them. Move the echoes and repeat. On a tunnel, an echo through the tunnel and another reached directly over the same link tell the tunnel's faults apart from the link's.
A sensor at 1000 packets every 5 minutes is a light watch. If it shows trouble, raise it to 10000 packets every minute for the hunt, and back again when the fault is found.
Running it by hand
The echo is its own client, so an engineer can run the same tests at a command line: oversight-echo -c 10.1.1.15 sends 1000 packets and then 1MB each way, and prints the same figures the sensors record. -udp 10000 and -tcp 100MB change how much; -udp 0 or -tcp 0 leaves that test out.
When it fails
| Error | What it means |
|---|---|
| REFUSED | The machine answered but nothing is listening on the port: the echo is not running, or is on another port. |
| RESPONSETIMEOUT | UDP: not one packet came back. The echo is not running, or a firewall drops the port, or the leg is down. TCP: the transfer stopped part way for longer than the response timeout; the message says how far it got. |
| CONNECTTIMEOUT | TCP: no answer at all to the connection, usually a firewall dropping the port or the machine being unreachable. |