Pushed logs: did the job work?

Pushed logs are the logs that automation writes about itself: a Proxmox backup job, a Proxmox Backup Server sync or verify, a Synology Hyper Backup task, a Virtualmin backup, a package update run, a script that imports leads. The job sends its whole log to Oversight when it finishes, and Oversight answers one question about it: did it work? A log that did not work goes to your rules, which raise a ticket, send an email or a Matrix message, or ring someone.

Most of these jobs already send a report somewhere, usually to a mailbox. The trouble is that nobody reads a mailbox of reports that say "OK" every night, and the report that matters looks almost the same as the rest: a subject line that still says the job finished, with one guest out of forty failed halfway down. Oversight reads every one, all the way through.

A job that can post sends its log over HTTPS. A job that can only send email sends it to Oversight by email, and it is read in exactly the same way: see Email in.

One log, one verdict

• Each log is judged alone. Nothing is remembered from earlier logs, merged or held back. A job that fails three nights running is three verdicts, and three actions.
• Worked, or did not work. There are no severities and no classes of failure. A log either shows the job did what it should, or it does not.
• In doubt, it did not work. A log the model cannot make sense of, or one it is unsure about, counts as not worked. A ticket that turns out to be nothing is a nuisance; a backup that has silently failed for a month is a disaster. A false alarm is put right by adjusting the channel's filter, below.
• The whole log is kept, exactly as sent, for 90 days, whatever the verdict. It may be the only copy there is.
• No log is ever expected. Many of these jobs run to no schedule, and the systems themselves are watched by sensors already. Only what arrives is judged, so a job that has stopped running altogether is a matter for a sensor, such as a file sensor on the backup it should have written.

Channels

Every log arrives on a channel, named by whatever sends it: proxmox-vzdump, pbs-sync, virtualmin. A new name makes the channel with its first log, so nothing is lost for want of setting one up. A log sent with no name goes to default. An account can have up to 100 channels; past that, a new name goes to default too.

• Several sources can share one channel. Three Proxmox clusters can all send their backup logs to proxmox-vzdump. Each log shows the host it says it came from, or else its address, and the Source filter picks one out.
• Each channel has its own filter and its own rules, so backups can raise a ticket while package updates send a Matrix message.
• Nothing is added to the tree. A channel is not a device or a sensor, and plays no part in rollups, availability or the SLA.

Channels are on the Logs page, for anyone holding the Logs permission. A push channel opens on the last 30 days, since its logs are few. Each log is one row, with its title, source and verdict. Open the title to read the whole log, with whatever came with it, or download it as a text file. The Day field answers "something seems to have failed on Tuesday, what did we get?", and the Text filter searches the whole of every log on the channel, not just the titles.

How a log is judged

Every log is read, in this order, and the first step that settles it decides:

1. A test notification. The Test button on a Proxmox VE or Proxmox Backup Server notification target sends a test message, which counts as worked. Only Proxmox's own exact test text passes as one, so nothing with more in it can.
2. The filter's notok tests. A pattern for a failure the filter already knows about: ERROR: Backup of VM 101 failed. A match is not worked, at once, and the model is never asked whether it might be fine.
3. The filter's ok tests. A pattern that shows success. Only reached when no notok test matched, so a known failure can never hide behind a line saying the job completed.
4. The model. Anything the filter has not settled, and every log on a channel with no filter, is read by a language model, which is asked whether the log shows that anything failed or did not complete as it should. If in doubt, the answer is that it did not work.

The model answers with its reason first, then the verdict, a short title and the exact lines it relied on. A line it quotes that is not in the log is flagged on the verdict, so a made-up line shows. Where a filter has already decided a log did not work, the model still reads it, to say what failed, where and why; the verdict stands whatever it says.

A channel with no filter is not a channel that is not checked. Tried with the default question alone, the model passed a complete Hyper Backup, a clean disk health report and a clean Virtualmin backup, and failed Active Backup's missed backups, a rejected lead and a Virtualmin log with one failure buried at server 31 of 63, whose last line still claimed all 63 had succeeded. A filter makes the answer quicker, cheaper and steadier. It is not needed to get one.

The model

• It runs in GEN's own datacentre. Logs are never sent to an outside AI service. Reading by the model is part of the account's AI service, which GEN switches on.
• The log is data, never instructions. It is given to the model marked as material to judge, with the instruction never to follow anything written in it. A notok test never goes near the model at all, so a log cannot talk its way out of a failure a filter knows about.
• A busy model is not a verdict. If the model is busy or unavailable, the log waits and is tried again a minute later. One that still cannot be read after four hours counts as not worked, with that as the reason, so it is looked at by a person.
• One at a time, oldest first, every minute. A log is usually judged within a minute or two of arriving.

Filters

A filter is optional, and lives on the channel's Filter tab. It does two things: it settles the logs it knows about without the model, and it tells the model what this kind of log is, so the model judges it better. It is one JSON document:

{
  "about":  "A Proxmox VE vzdump job log. It opens with a table of guests, each ok or err, then each guest's log. A good one lists every guest as ok.",
  "notok":  [{"re": "ERROR: Backup of VM \\d+ failed", "title": "A guest failed to back up"},
             {"field": "severity", "is": "error", "title": "Proxmox reported an error"}],
  "ask":    "Did every guest produce a backup, with nothing failed or skipped?",
  "look":   {"exclude": ["^\\d+: \\S+ \\S+ INFO:\\s+\\d+% \\("]}
}

Patterns ignore case, and ^ and $ match at the start and end of each line. A filter is checked when it is saved: valid JSON, known keys, and every pattern must compile. A pattern that fails while a log is being read is shown on the tab as Pattern failed, and the log goes on to the model rather than going unjudged.

The library

Ready-made filters, to use as they are or to start from. Choose one and press Use this, and it is copied onto the channel. It is a copy: from then on it is the channel's own, to change as you like, and GEN improving the library never changes what your channel does.

• Proxmox VE: vzdump backup
• Proxmox: package updates (apt)
• Proxmox Backup Server: sync
• Proxmox Backup Server: garbage collection
• Proxmox Backup Server: verify or prune
• Synology: Hyper Backup
• Synology: Active Backup for Business
• Synology: disk health
• Virtualmin: backup
• Lead ingest
• Start from nothing

A filter of your own can be kept in your account's library with Keep, and then appears in the list as Yours, for your other channels.

Below the filter, How it treats the latest pushes runs the saved filter over the channel's last ten logs and shows, for each, Worked, Did not work or Asks the model. The model is not called and nothing is decided or sent, so a change can be tried against real logs before it is relied on.

Anyone holding the Logs permission can read a filter. Changing one needs the Log filters permission, ticked per user on Setup > Users. Every save is recorded in the audit log, with the filter as saved.

Worked example: Proxmox backups

A vzdump job's log opens with a table of guests, then each guest's own log:

VMID  Name      Status  Time      Size
101   dc01      ok      00:04:12  38.2 GiB
102   files     err     00:00:03  0 B
103   web       ok      00:01:55  12.7 GiB

The library filter's notok tests catch err in that table and Proxmox's own ERROR: Backup of VM 102 failed line, so this log is not worked without the model being asked, and the model then reads the log to say what went wrong with guest 102. The look exclusion keeps the model from reading thousands of progress lines; the stored log keeps them.

Worked example: a failure buried in the middle

Virtualmin writes a section for each virtual server, then a total. A failure on one server sits in the middle, and the total at the bottom can still claim every server succeeded. A pattern on the last line would pass it. So the library's Virtualmin filter has no ok test at all: it tells the model what a good section looks like (.. done, .. none defined) and asks whether every server backed up and uploaded, and the model reads every section. A filter should leave out ok wherever a good line can sit below a failure.

Putting a false alarm right

Since a doubtful log is not worked, false alarms happen, especially on a new channel. Each is fixed once, on the channel's filter:

• A warning the job always gives and that does not matter: say so in prompt, as in "The iothread warnings are known and of no interest."
• Lines that confuse the model and carry nothing: leave them out with look.
• A log that always looks the same when it has worked: an ok test, as long as no failure can sit above the line it matches.

When a log did not work

The channel's rules decide what happens. Make a rule in Actions with CRIT in its states, then bind it on the channel's Filter tab, either to this channel alone or to every push channel on the account, including those not made yet. One rule has one method, so a ticket and a Matrix message are two rules. Rules keep their schedules, escalation and acknowledgement, as for any alarm.

• One action per log. Each log that did not work is told once, on its own.
• The message says what failed. It opens with the model's account of what failed, where and why, then the source, the lines relied on and a link to the log on its channel. The whole log goes with it as a text file, so a ticket carries the log too.
• Held, never dropped. A rule that has reached its message limit holds a failed log until it has room, rather than losing it: the log may be the only word that something failed. Each channel has an allowance of its own, apart from the estate's alarms, so a fibre link going down cannot use up the allowance for a failed backup.
• No rule, and GEN is told. The Filter tab shows, in red, when no rule covers the channel. A log that does not work on such a channel alerts GEN's staff, so it is still seen.

Things to know

• Cost. Reading by the model is charged in credits, by how much the model reads, so a long backup log costs more than a short one. A log settled by a filter alone costs nothing, and a second try after an unusable answer is not charged.
• Personal data. Some logs carry it: a rejected lead arrives with a person's name, address and phone number. It is kept in full for the 90 days, shown to users holding the Logs permission, read only by the model in GEN's datacentre, and goes with any ticket raised.
• Up to 8 MB a log. A Proxmox backup log for a busy cluster is typically under 100 KB.