Everything Oversight watches is described as one tree, the estate. It is built under Setup, Configuration, where the Estate panel on the left is the tree itself and the panel on the right is whatever is selected in it. The same tree drives the dashboard, the state of every object and where notifications apply.

The levels

A device cannot sit directly on a site, so every new site is given a group called Default to hold its first devices. Rename it, or add groups of your own beside it. Names must be unique among the objects sharing a parent.

Probes are not part of the tree. They are the machines, run by GEN, that make the checks, and a sensor chooses which probe groups watch it. The tree describes what is monitored; probe groups describe where it is watched from.

Adding to the tree

Press Add on the Estate panel. The dialogue suggests the next level down from whatever is selected, so with a group selected it offers a device under that group. Choose the type, where it goes under, and a name. A device also needs its hostname or address. A sensor needs its type, an interval and at least one probe group, and can start from a template; its name may be left empty, in which case it is named after its type.

Objects cannot yet be moved to a different parent. To move a device, save its sensors as a template, apply the template to a new device in the right place, and delete the old one.

States

UNKNOWN and STALE are never treated as down. They are shown plainly as no data, they never drive a parent into alarm, and they are kept apart from downtime in availability figures. Equally, they are never counted as healthy. HELD is treated the same way: the downtime belongs to whatever it depends on, and that is what alarms.

ABNORMAL and UNUSUAL are the opposite case: the thing is working, and counts as up everywhere, but is not behaving as it normally does. Both are quieter than a warning and notify nobody by default.

On the dashboard: the wheel

The dashboard draws the same tree as a wheel. The object in the middle is the one in focus, and each ring outwards is one level further down: sites, then groups, then devices, up to four rings. Sensors are not drawn as a ring; a device is the end of its branch, and its sensors are listed when it is selected. Every segment is coloured by its state, so a problem shows as a coloured patch on the outer ring with the path to it running inwards.

Click a site or group to centre the wheel on it; the button in the middle, named after the level above, goes back up. Click a device to select it and see its sensors and readings. The tree on the left is the same estate again, and it opens by itself wherever something is not OK, so what needs looking at is never folded away. A paused or scheduled object is blue, and so is everything beneath it. The view is kept in the address bar, so a link to it opens the wheel where it was left.

Rollup: how state travels up the tree

Every object from a sensor upwards has a Rollup panel. It decides the object's state from what sits beneath it, and the same three questions are asked at every level:

Only those beneath with an opinion are counted, meaning those at OK, WARN, CRIT, ABNORMAL or UNUSUAL. Anything UNKNOWN, STALE, HELD, paused, suspended or outside its schedule is left out of the count altogether. Of those counted, anything at WARN or CRIT is failing, and the number failing falls into one of three buckets:

Each bucket is mapped to OK, WARN or CRIT, and that is the object's state. If nothing is failing the state is ABNORMAL if anything counted is ABNORMAL, otherwise UNUSUAL if anything is UNUSUAL, otherwise OK. Two refinements keep the verdict honest:

• Minimum reporting. Below this many with an opinion, the state is UNKNOWN rather than a verdict reached from one straggler. A sensor watched from three probe groups with a minimum of two stays UNKNOWN while only one of them is heard from.
• The mapping can soften, never inflate. If a bucket maps to CRIT but nothing beneath is actually CRIT, the result is WARN. A handful of warnings never becomes a critical.

Because the buckets count, the arithmetic has two consequences worth knowing. With a single probe group or a single child, one failure is ALL, not ANY. With exactly two, one failure is ANY and two is ALL, so SOME can never apply.

The defaults

A sensor shrugs off one probe group struggling, since that is usually the path and not the target. A device with one failing check needs a look, and with more than one is in trouble. A group or site treats one failing member as fine, several as a warning and all of them as critical. Set the mapping to suit the thing: a standalone server should go red on any failure, while twenty web servers behind a load balancer only matter when several fail. Keep each mapping at least as bad as the one before it, so that a second failure can never improve a state.

Working through Figure 1 with the defaults:

• web2 has two of its three sensors failing, and at least one is CRIT. Two is SOME, which on a device maps to CRIT, so web2 is CRIT.
• nas1 has one sensor at WARN. One is ANY, which on a device maps to WARN.
• Rack A holds only nas1, so its one failing device is ALL, which maps to CRIT. Nothing beneath it is actually CRIT, so the mapping softens and Rack A is WARN.
• Rack B holds nas2 and nas3. nas2 is paused and left out, and nas3 is healthy, so Rack B is OK.
• Web cluster has one failing device out of three. One is ANY, which on a group maps to OK: one web server down behind a load balancer is not the service down. web3's UNUSUAL does not colour the group, because something beneath is failing and the mapping has already decided.

A group reading OK does not hide web2. It is still CRIT in its own right, it is in the alarm list, and a rule bound to the group and everything beneath it is still told. Rollup decides what each level says about itself, not who hears about what.

Contributes upwards

Clear Counts towards its parent's state to watch something without letting it condemn what it sits on. A certificate expiry warning is worth seeing on its own, and is not the device being down. Clear Counts towards availability reporting to keep an object out of availability figures.

Inheritance

Some settings are made once, high in the tree, and reach everything beneath. The nearest one always wins.

Interval, timeouts and probe groups are set on each sensor and are not inherited. Templates are the way to set them once for many sensors.

Pausing, schedules and suspension

Pause is at the top of any site, group, device or sensor. Choose how long, from five minutes to indefinitely, and give a reason, which is shown beside the object and recorded in the audit log. A timed pause lifts itself when the period is up; Resume lifts it at once. Nothing beneath a paused object is polled, and none of it counts towards an alarm. On the dashboard it all shows blue.

An object outside its schedule is marked SCHEDULED OFF. It is not polled, any results that arrive are discarded, and it is left out of its parent's state. When the schedule opens again it is given time to report before it can read as stale.

SUSPENDED is set by Oversight, not by you. A sensor whose settings a probe cannot run is suspended after a single attempt and set to UNKNOWN, rather than failing over and over and waking somebody. A banner on Configuration lists every suspended sensor. Correct the sensor and save it, and it is collected again on the probe's next configuration poll. Lift all clears every suspension at once; any sensor still broken is suspended again on its next attempt.

Deleting

Delete removes the object and everything beneath it, and the button shows how many objects that is. The sensors' settings, and any credentials held on the deleted objects, go with it. It cannot be undone, and the tenant itself cannot be deleted.