Phone system monitoring from GEN

The trunk dropped at 07:00.
You knew by 07:03.

Oversight watches any phone system that speaks SIP, Asterisk, FreePBX and 3CX included, from as many places as you choose. When a SIP trunk or a carrier stops answering, you get one message that names it, before your first caller finds out.

ALARMPhones critical · 2 new 07:03 UTC
Carriers                          CRIT
└ sip.carrier.example             CRIT  3m
  └ SIP OPTIONS  RESPONSETIMEOUT  CRIT
Head office                       WARN
└ pbx-01                          CRIT  3m
  └ PJSIP  trunk offline          CRIT
Before the office opens

What 07:00 looks like.

07:00The carrier stops answering. Calls to your main number fail, and the office is empty.
07:01The first checks fail. One failure, from one place, alarms nobody.
07:03The third failure in a row, seen from several places. One message names the carrier and the trunk. Your phone system's own SIP check is not in it, because it is still answering, so you already know where to look.
07:03The same rule calls your webhook, and your script moves the inbound numbers to the backup carrier. It could acknowledge the alarm once they have moved; here it leaves that to a person, since the carrier is still down.
07:13Nobody has acknowledged it, so the alarm climbs to a phone call, read aloud to whoever is on call.
07:14They acknowledge it from their phone, and every tier stops.
09:00Reception opens to working phones.
An illustrative morning. The timings are settings: carrier and trunks checked every minute, three failures in a row, ten minutes to escalate.

SIP, not a port test

A probe sends SIP OPTIONS over UDP, TCP or TLS and reads the reply. Nothing rings and no call record is left, and a system whose SIP stack has wedged is caught even while its port still accepts connections.

Setting it up

Carrier or you?

Watch the carrier's SIP endpoint as a check of its own, with the probe's address allowed if it only answers known peers. When a carrier fails your phone system can still be reached, so its trunk still alarms, in the same message as the carrier. A dependency is for the site's internet connection: when that fails it alarms once, and the checks that can no longer reach the phone system are held rather than alarmed.

How dependencies work

The webhook fix

A rule can post to any URL, with the method, headers and body yours to template. Your script does the repair, such as moving numbers to another carrier, and can acknowledge the alarm through the API so nobody is woken for something already dealt with.

Rules and channels

Escalation to voice

If nobody acknowledges, the alarm climbs after the minutes you set, to a second team or a phone call read aloud. The call is placed through GEN's phone system, so it still arrives when yours is the thing that has failed.

How escalation works

Any SIP phone system

What Oversight reads on yours.

Asterisk

SIP OPTIONS from a probe, then ARI, Asterisk's REST interface, through a read-only user. Every PJSIP endpoint becomes its own reading, so an alarm names the trunk, and a trunk you have named that is deleted or renamed reads as missing rather than quietly vanishing. Security events are counted per hour from syslog, so a botnet at the door shows as a spike.

ARI cannot see a trunk's outbound registration, so a SIP check against the provider covers that.

Asterisk and FreePBX guide

FreePBX

The same three readings, set up in the interface with no module to install: the mini-HTTP server in Advanced Settings, a read-only ARI user, and the probe's address as Trusted in the firewall, so that it is not banned after a few polls. The logging line goes where Apply Config will not overwrite it.

Asterisk and FreePBX guide

3CX

SIP OPTIONS from a probe shows whether the SIP stack is answering, and an HTTP(S) check watches the web client and its certificate. There is no ready-made reading of the 3CX API yet: tell us which version you run and we will work it out with you.

Talk to GEN

Anything else: if it speaks SIP we can watch it, and if it has a REST API we can interrogate it.

The machine underneath

A phone system is also a server.

A healthy trunk is no comfort on a failing disk. All of this runs on the same platform, with the same rules, messages and escalation.

Disks and filesystems

The Linux agent judges the host on the host: a software RAID array degraded or rebuilding, a disk offline or slow, a filesystem read-only, full or due to fill within twelve hours. It answers reads and can change nothing.

The Linux agent

Network, CPU and memory

TCP retransmissions, interface errors, lost links, a port renegotiated slower and a full connection tracking table, which silently drops new connections. CPU, load and memory are charted over time, so choppy audio on Tuesday afternoon can be checked against what the server was doing.

What the agent checks

Syslog, screened on site

A syslog sender at the site takes the phone system's syslog, and rules there pick out the lines that matter, so only those leave. Failed registrations and passwords are counted per host, trended, and included in scheduled reports.

The syslog sender

Backups and Asterisk's own output

A backup job's log can be posted or emailed in, read to the end and judged on whether it worked. The log sender follows any file or reads stdin, so the output of a command such as asterisk -rx "pjsip show registrations" can be piped in and screened by your own rules.

Pushed logs and the log sender

Calls per hour

The database sensor runs one statement of yours, under an account that can only SELECT, and alarms on the answer. Against FreePBX's call records: SELECT COUNT(DISTINCT linkedid) FROM asteriskcdrdb.cdr WHERE calldate > NOW() - INTERVAL 60 MINUTE. A count that falls to nothing in office hours is a fault no SIP check will see.

How sensors work

Updates pending

The package agent shows what is waiting on each host, finds a package across every phone system in one search and patches them in one action, leaving alone anything you protect. It covers operating system packages on Debian-based systems such as FreePBX 17 and 3CX, not FreePBX modules.

The package agent

Pay for what runs

A phone system, watched properly, for about £9.68 a month (roughly $13 or €12*).

No per-sensor licence, no tiers and no prepayment. Invoiced monthly in arrears, in GBP. *Estimated: other currencies vary with the exchange rate.

For a monthCost
SIP OPTIONS every five minutes, from three locations£1.04
Endpoints and trunks through ARI every minute, from a probe on site£6.91
The Linux agent's verdict every minute, from a probe on site£1.73
  • The carrier's own SIP check, every minute from three locations, is £5.18, shared by every phone system on that carrier.
  • Rules, escalation, email, webhooks and tickets are included. SMS and voice are metered separately.
  • Rather GEN set it up? See services, from £49 an hour.

Find out from Oversight, not from your callers.

Register as a client and an account manager will walk you through the early steps, with no commitment at all. Or talk to an expert first.

The same platform watches the rest of the estate too, from Proxmox and Ceph to Windows servers and email. See what else it covers.