Proxmox VE: guests and cluster health

Proxmox VE answers for the whole cluster from any one of its nodes, through the same API its web interface uses on port 8006. Two HTTPS sensors on one node's device read it, each answering a different question:

Both need one thing on the Proxmox side, a read-only API token, and one credential in Oversight. Neither installs anything on the nodes.

A read-only token on Proxmox

Give Oversight a user of its own with the built-in PVEAuditor role, which can read the state of everything and change nothing, and a token on that user. Run this on any one node, as root; a cluster shares its users, so it applies to every node:

pveum user add oversight@pve --comment "Oversight monitoring"
pveum acl modify / --users oversight@pve --roles PVEAuditor
pveum user token add oversight@pve monitor --privsep 0

The last command prints the token's full-tokenid, here oversight@pve!monitor, and its value, a long string of letters and digits. The value is shown this once and cannot be read back later, so copy it before closing the terminal.

The same can be done in the web interface: add the user under Datacenter > Permissions > Users, give it the PVEAuditor role on path / with Propagate ticked under Datacenter > Permissions, then add a token under Datacenter > Permissions > API Tokens with Privilege Separation unticked.

Unticking privilege separation lets the token act with its user's permissions, which here are read-only anyway. Left ticked, the token has no permissions of its own until it is given the same role on / itself, and until then the guest list comes back empty rather than refused. The pve realm keeps the user inside Proxmox, with no Linux account on any node.

The credential in Oversight

Add an HTTP(S) credential with the full token id, oversight@pve!monitor, as the Username and the token's value as the API key. Both templates send them as Proxmox expects, in the header:

Authorization: PVEAPIToken={{basicuser}}={{apikey}}

Hold the credential on the group the cluster's nodes sit in rather than on one device, so every node beneath it inherits it. See Credentials for how a credential is found.

A Proxmox node presents a self-signed certificate unless it has been given a proper one, so both templates have certificate verification off. The connection is still encrypted. If your nodes have certificates from a public authority, through Proxmox's own ACME support for example, turn Verify TLS on.

Which node to point at

Add both sensors to the device for one node, with that node's address. Any node answers for the cluster, so there is no need to put them on every node, and doing so only raises the same alarm several times.

The node you choose is the one thing these sensors cannot see past. If it goes down, both fail to connect and go critical after three failed polls, three minutes as shipped, without being able to say anything about the rest of the cluster. Give both a dependency on that device's Ping, so a node that has gone raises one alarm rather than three; see Dependencies. For a cluster that matters, a second Proxmox Cluster Health sensor on another node's device keeps the cluster in view while the first is down.

Guests: the Proxmox Guests template

The template asks for /api2/json/cluster/resources?type=vm once a minute, which lists every VM and container in the cluster with its state and the node it is on, and passes the answer through the uf_pveguests user function. That turns it into one reading per guest, keyed by its VMID:

A lock wins over the run state, so a guest being backed up reads 2 whether or not it is running. One left locked after a failed backup reads the same way until someone clears it with qm unlock or pct unlock, which is worth an alarm in itself, since Proxmox refuses to manage a locked guest.

Setting it up

1. On the node's device, add a sensor from the Proxmox Guests template.
2. Replace its extraction rows with your own guests. The template comes with GEN's as an example: each row reads one guest by its VMID as the path, 114, and is named after it, 114-osticket, so the alarm says which guest without anyone needing to look the number up. The rows start at SV03, since SV01 and SV02 are the response's status code and size.
3. Replace the user function's arguments with the same VMIDs: {"expect": [114, 133, 102]}. These are the guests that must exist. One missing from Proxmox's list then reads 8 rather than having no reading at all, because a rule whose reading is missing is skipped, and a deleted guest would otherwise read as OK.
4. Give each row a rule. As shipped, every guest that is not 1, 2 or 3, that is not running, backing up or migrating, is critical: a rule of not equal to (1,2,3). A guest that matters less can take a warning instead.

The template also carries a rule of status code not equal to 200 for critical. Keep it. A token that has been deleted or has expired gets a 401, which leaves no guest readings at all, every guest rule is skipped, and without this rule the sensor would read OK while watching nothing.

Every poll also carries notrunning, the guests that are not running with their names and nodes, such as 133 proxy stopped on nebula. Read it into a text slot if you want the whole list beside the readings.

Many guests: groups

A sensor has room for 48 guests. Past that, or where guests fall naturally into sets, name groups in the arguments instead:

{"groups": {"core": [114, 133, 102], "infra": [105, 117]}}

Each group gives a count of its members that are not running, backing up or migrating, read with the path core, and their names, with the path core_down. One row and one rule of greater than 0 then covers a whole group. Group members are expected, so one that has been deleted counts against its group. To treat a guest mid-backup as down too, add "ok": [1].

The cluster: the Proxmox Cluster Health template

The template asks for /api2/json/cluster/status once a minute, which is corosync's own view of the cluster: its name, whether it is quorate, how many nodes it should have and which are online. The uf_pvecluster user function turns that into:

The count of nodes comes from corosync rather than from the list, so a node missing from the list altogether still counts as offline rather than quietly shrinking the cluster.

The rules as shipped

Change the last rule to your own number of nodes. It is there because removing a node with pvecm delnode shrinks corosync's count as well, so the cluster is fully healthy again with one node fewer, and nothing else would notice. The rule makes a change of membership something somebody decides on purpose rather than finds out later.

What a split looks like

The answer depends on which side of the split the polled node is on. On the side that keeps quorum, the cluster reads quorate with the missing nodes offline: a warning naming them. On a side that has lost quorum, it reads not quorate: critical. A node the probe cannot reach at all reads as a failed connection, which is why the dependency on its Ping matters.

A single Proxmox node that is not in a cluster has no quorum to lose and no membership to change. Its healthy, quorate and cluster readings are left empty and their rules skipped, so the cluster sensor adds nothing there: the guests sensor and a Ping cover it.