Behaviour: when a sensor is working, but not as it usually does

A threshold says something has broken. Some things never break a threshold on their way to failing: a router whose memory creeps up for three weeks passes every check anyone set, right up to the day it falls over. Others are working perfectly well but doing something they never normally do, such as a mail queue at twice its usual size on a Tuesday afternoon. Neither is an alarm, and both are worth knowing about.

Oversight has two states for these. Both mean the sensor is working, but not behaving as it normally does:

They answer different questions. ABNORMAL is the one that gets ahead of a failure. UNUSUAL says something changed today, which is as often a customer's workload as a fault.

Switching it on

Open the sensor and tick Statistical analysis. It is off by default, and it is worth setting on a handful of sensors rather than across the estate. Analysis run over everything gives everything an opinion, and people soon learn to ignore it. Choose the sensors whose drift you would act on: memory and disk on the servers that matter, latency on the circuits you pay for, queue lengths, temperatures.

From then on, every reading the sensor takes is kept in the statistics store as well as in its history, and that is what both states are judged from. What is analysed:

• Response time and connect time, for every sensor that measures them.
• Every numeric value the sensor extracts, in the units it is charted in. A counter read as a rate is analysed as the rate.

Text values are not analysed, since they have no average to depart from. Nor is anything that counts elapsed time, such as an uptime: it rises at every reading by definition and drops to nothing at every restart, so it would always be going somewhere. Oversight recognises these from the readings themselves and leaves them out.

UNUSUAL: not like other Tuesdays

Every quarter of an hour, each analysed value is compared with the same quarter hour on the same weekday in up to 20 earlier weeks. Today at 14:15 is compared with 14:15 on previous Tuesdays, not with 03:00, and not with Saturday.

A single odd quarter hour is not enough. A sensor becomes UNUSUAL when four quarter hours in a row are out of the ordinary in the same direction, so an hour at twice the usual level is a finding and a ten-minute spike is not. It clears by itself when the value comes back, and one ordinary quarter hour in the middle of an episode does not clear it, so a busy afternoon with a lull in it is one finding, not three.

A quarter hour in which more than half the readings are missing is not judged at all: a level worked out from two readings out of fifteen is not evidence of anything.

The transition says what was seen, for example:

Messages queued higher than usual for the time of week: 1,040 against 512, for 1 h 15 min

ABNORMAL: going somewhere

Each night, after the day has finished, each analysed value is looked at over several lengths of time at once, each compared with the periods of the same length before it:

A sensor becomes ABNORMAL when two of these have moved in the same direction and none has moved the other way. Up on the day and down on the month is a sensor oscillating, not one trending. The day on its own counts only after three days in a row have moved the same way, because one bad day nudges the week as well, and a single bad day must not count as two pieces of evidence.

This is what makes a slow change visible. A creep shows on the day first, then on the week, then on the month, and every reading on the way may be individually unremarkable. A spike moves one period and reverts, and says nothing.

The transition gives the breakdown, which is what a report wants to say, for example:

Memory used rising: up 12% on the week (71% against 63.4%) and up 9% on the month (69.8% against 64%); level on the weekday

Periods without enough history yet are listed as too early to judge, so it is always clear what a finding was built from.

How far is too far

There is no fixed figure, no three standard deviations and no twenty per cent. A fixed figure is too tight for a sensor that is naturally restless, which then reports all day, and too loose for a steady one, which then never reports at all. Two ping sensors, one across a leased line and one across home broadband, have nothing in common.

Instead, each sensor's own history decides. For every period, Oversight looks at how far that sensor's past periods sat from the ones they were compared with, and a change has to be larger than all but the most extreme of those before it counts. The limit is worked out separately for each value, each period length and each probe, and it is worked out again every time, so it follows the sensor as its history grows.

Each period is judged by three figures:

• The average, which sees everything.
• The median, which ignores a short burst.
• The floor, the lowest tenth of the readings: what the thing does when nothing is being asked of it. A creeping problem often shows here first, while the busy hours still look the same.

At least two of the three must agree before the period counts as having moved. A change of less than a fiftieth of the usual level never counts, however steady the sensor has been.

Every probe has its own baseline

A sensor watched from two probes is two sets of readings, and each is judged on its own: a ping from London and the same ping from Frankfurt have nothing to say about each other's normal. Where a sensor has more than one probe, the transition names the one that saw the change.

This is useful in its own right. If several sensors on different devices all become ABNORMAL from one probe and not from another, the change is in the probe or its host, not in what it is watching.

How long before it says anything

Nothing is reported until there is enough history to judge against, and that is deliberate. A baseline from a few days is a guess, and a guess here would put cyan and teal across the dashboard for no reason. The clock starts when analysis is switched on:

Since ABNORMAL needs two periods to agree, it can first be reported after about five weeks. Its judgement gets steadier as the longer periods join in.

What it does to everything else

• A real alarm always wins. A sensor that is WARN or CRIT shows that, whatever the analysis thinks. ABNORMAL is shown in preference to UNUSUAL, because a trend is the part nobody else will catch.
• It counts as working. Both states are up in availability, in reports and in the API, and both count as reporting in a rollup.
• It colours what is above it. A device, group or site with nothing failing beneath it takes ABNORMAL if anything beneath it has it, otherwise UNUSUAL, otherwise OK, the same way one failing sensor colours its device. See Rollup.
• It notifies nobody. A notification rule cannot be set to either state. A sensor that goes from CRIT to UNUSUAL has stopped failing, so anyone told of the alarm is told it has cleared, as they would be for OK.
• It never sounds.

Things to know

• Both directions count. A response time that halves is reported as well as one that doubles, and a value recovering after a long climb is a movement too. There is not yet a setting for a sensor that only cares about one direction.
• Changing what a value means looks like a change. Editing a value's multiplier, its unit or what it extracts changes the number from that reading onwards, and the history behind it was measured the old way. Expect the analysis to notice, and to settle again once the new figures have built up.
• Switching analysis off ends any finding at once. Nothing is collected while it is off, so switching it back on leaves a gap in the history, which is judged around rather than as a change.