How to monitor particular kit with Oversight: which sensor to use, where on the device to set up access, which user function and extraction rows read the response, and how to write the rules that turn it into an alarm. Search for a vendor, a product or an error message, or ask a question in your own words.
HPE iLO and Dell iDRAC: server hardware through Redfish
A server's management controller, iLO on HPE and iDRAC on Dell, is a small computer of its own inside the chassis. It watches the hardware whatever the operating system is doing, so it can report a failing power supply, a stopped fan or a faulty array controller on a server whose software looks perfectly healthy, and it still answers when that software has crashed.
Both speak Redfish, an open standard from the DMTF: JSON documents over HTTPS. Oversight needs nothing special to read it and nothing is installed on the server. An ordinary HTTP(S) sensor asks for a document, extraction rows pull out the readings, and rules decide what is wrong.
| Question | How |
|---|---|
| Does the server think anything is wrong with it? | The iLO Health template. Start here: it is the one sensor that catches faults you did not think to look for. |
| How hot is the room, and the processors and power supplies? | The iLO Temps template. |
| What about the array controller, network cards and exhaust, and is anything near its limit? | The iLO Sea of Sensors template, named after HPE's own term for the server's many temperature sensors. |
| How much power is it drawing, and are both power supplies sound? | The iLO Power template. |
The templates and every iLO reading in this article were tested against iLO 4 firmware 2.82 on ProLiant Gen8 and Gen9 servers. Other generations and iDRAC differ in ways set out under Other generations, and Dell iDRAC. Those details come from the vendors' published guides and have not been run here, so check them against a stored response before relying on them.
What is available
Redfish is a tree of documents, each of which links to the ones beneath it. Every path below starts /redfish/v1/, which is itself the top of the tree. Keep the paths exactly as shown, trailing slash included.
| Document | iLO | iDRAC | What it holds |
|---|---|---|---|
| System | Systems/1/ | Systems/System.Embedded.1 | Overall health, power state, model, BIOS version, and a health summary for memory and for processors. |
| Chassis | Chassis/1/ | Chassis/System.Embedded.1 | Health of the enclosure, and links to the next two. |
| Thermal | Chassis/1/Thermal/ | Chassis/System.Embedded.1/Thermal | Every temperature with its limits, and every fan. 37 temperatures on a DL360 Gen9, 49 on a DL560 Gen8. |
| Power | Chassis/1/Power/ | Chassis/System.Embedded.1/Power | Power drawn, with the average and peak over the last 20 minutes on iLO, and each power supply's health, output and input voltage. |
| Storage | Systems/1/SmartStorage/ | Systems/System.Embedded.1/Storage | Array controllers, then logical and physical drives beneath them, one document each. |
| Memory | Systems/1/Memory/ | Systems/System.Embedded.1/Memory | A list of the fitted memory modules, each with a document of its own. |
| Processors | Systems/1/Processors/ | Systems/System.Embedded.1/Processors | The same for processors. |
| Manager | Managers/1/ | Managers/iDRAC.Embedded.1 | The controller itself: firmware version, licence on iLO, self-test results. |
| Logs | Systems/1/LogServices/IML/Entries/ and Managers/1/LogServices/IEL/Entries/ | Under LogServices | The hardware log and the controller's event log. See Logs. |
Setting it up
A device for the controller
Add the controller as a device of its own, with the controller's address, not the server's. It is a separate computer on a separate network connection, so losing it says nothing about the server, and losing the server's operating system says nothing about it. The sensor's probe must be able to reach the management network, which is often kept apart from everything else. Give the device a Ping and make the Redfish sensors depend on it, so a controller that drops off the network raises one alarm rather than several; see Dependencies.
A login that can only read
Give Oversight a local user of its own. On iLO, add it under Administration > User Administration with the Login privilege and every other privilege unticked. On iDRAC, add it under iDRAC Settings > Users with the Read Only role. Do not use an administrator's account: an account that can only read cannot power a server off, even if its password gets out.
In Oversight, add an HTTP(S) credential with that username and password. The sensors send them in a header:
Authorization: Basic {{basicauth}}
A sensor uses the nearest HTTP(S) credential above it in the tree. Where the controllers sit in a group that also holds a credential for something else, such as a Proxmox token, hold this one on each controller's device or on a group of its own. See Credentials.
Controllers usually present a self-signed certificate, so the templates have Verify TLS off. The connection is still encrypted. Turn it on if yours have proper certificates.
How often to poll
The templates poll every five minutes rather than every minute, which matters more here than on most sensors, for two reasons. Keep to it unless you have a reason not to.
- Every request is a fresh login, and iLO records each one in its event log as
Browser login. With two sensors polling each minute, one iLO 4 was left holding a little over three hours of event log, so a real login by a person, or an attack on the account, soon scrolls out of it. - Controllers are slow. iLO 4 usually took one to two seconds to answer and just over five at worst, which the template's ten second response timeout allows for.
Sensors on a device making an identical request share one fetch, so several sensors reading the same document cost one login between them.
The iLO Health template
This is the most useful single sensor. The controller rolls everything it knows into one word in the system document, so you do not have to list each thing that could fail. The first time we read it on our own ProLiants it showed warnings in their storage that no temperature or power reading had given any sign of.
It asks for /redfish/v1/Systems/1/ and reads these into text slots, starting at SD03 since SD01 and SD02 are the response's status text and content type:
| Slot | Reading | Expression | Reads |
|---|---|---|---|
SD03 | Health | Status.Health | OK, Warning or Critical |
SD04 | Power | PowerState | On or Off |
SD05 | Memory | MemorySummary.Status.HealthRollUp | OK on iLO 4 |
SD06 | Processors | ProcessorSummary.Status.HealthRollUp | OK on iLO 4 |
SD07 | Memory | MemorySummary.Status.HealthRollup | Later iLO |
SD08 | Processors | ProcessorSummary.Status.HealthRollup | Later iLO |
SD09 | Model | Model | ProLiant DL360 Gen9 |
The memory and processor rows appear twice because paths are exact, capital letters included. HealthRollUp is how iLO 4 spells it, and the standard, and so later iLO, spells it HealthRollup. Whichever your controller does not use stays empty and its rule is skipped.
| Rule as shipped | State | Catches |
|---|---|---|
Status code not equal to 200 | CRIT | The login refused or the document missing. Keep this one: without it a wrong password leaves every other slot empty, their rules are skipped, and the sensor reads OK while seeing nothing. |
Health equal to Critical | CRIT | Anything the controller calls critical. |
Health not equal to OK | WARN | A warning, and any word this list does not expect. |
Power not equal to On | WARN | The server is off, with its controller still answering. Remove this for a server that is meant to be off at times. |
Memory not equal to OK | WARN | Names memory in the alarm. |
Processors not equal to OK | WARN | Names processors in the alarm. |
The sensor may go amber as soon as you save it. That is the sensor working, not a fault in the setup: the server has been reporting this all along, and the controller's own web page should show the same state. Two things follow. Health says that something is wrong, not what, so see the next section. And a warning left unfixed keeps the sensor amber, so a second fault of the same level changes nothing and raises no new alarm: clear them as they come, or watch the part that matters most with a sensor of its own.
Finding out what is wrong
The controller's System Information page is the quickest way. To have it in the alarm, read the part concerned:
| Where | Document | Expression |
|---|---|---|
| Memory or processors | System | The two summary rows above. |
| Storage as a whole | Systems/1/SmartStorage/ | Status.Health |
| One power supply | Chassis/1/Power/ | PowerSupplies.0.Status.Health, then .1 |
| One fan | Chassis/1/Thermal/ | Fans.0.Status.Health, and so on |
A second sensor on Systems/1/SmartStorage/ is worth having: array controllers are a common source of warnings, a failed cache battery being the classic case. Its response is small and it says which part to look at.
An item the server reports as absent carries a State of Absent and no Health at all, so a rule on its health is skipped. Where a part going missing matters, as a power supply does, test PowerSupplies.0.Status.State as well, not equal to Enabled.
Individual drives
Each physical drive is a document of its own, and each costs a request. iLO 4 ignored both of the usual ways of asking for several at once, $expand and $select, and answered with the plain document. The path also depends on the controller: an embedded one is ArrayControllers/0/, but one on a card can sit at its slot number, such as ArrayControllers/2/, and 0 then answers 404. Look at Systems/1/SmartStorage/ArrayControllers/ to see which yours has.
A drive at ArrayControllers/0/DiskDrives/0/ gives Status.Health, Location such as 1I:1:1, and DiskDriveStatusReasons.0. Its temperature fields read 0 on the SAS drives we tested, so they are of no use there. For most estates the storage sensor above is the better bargain: it covers every drive for one request.
How a template reads a document
All four templates share one request, differing only in the path:
| Setting | Value |
|---|---|
| Path | /redfish/v1/Chassis/1/Thermal/ for this example |
| Method | GET |
| Headers | Authorization: Basic {{basicauth}} |
| Scheme and port | https, port 0 for the default |
| TLS | Verify off, minimum 1.2 |
| Body cap | 65,536 bytes |
| Timeouts | 5 seconds to connect, 10 to respond |
| Poll and failure threshold | Every 5 minutes, 3 failed polls |
The Thermal document looks like this, cut down to one of each:
{"Temperatures": [
{"Name": "01-Inlet Ambient",
"ReadingCelsius": 21,
"UpperThresholdCritical": 42,
"UpperThresholdFatal": 46,
"PhysicalContext": "Intake",
"Status": {"Health": "OK", "State": "Enabled"}}, ...],
"Fans": [
{"FanName": "Fan 1",
"CurrentReading": 19,
"Units": "Percent",
"Status": {"Health": "OK", "State": "Enabled"}}, ...]}
An extraction path is names separated by dots, with a number to pick from a list, counting from 0. So the first temperature's reading is Temperatures.0.ReadingCelsius. That is simple, and for temperatures it is wrong, as the next section shows.
Temperatures: why not by position
The list is not the same from model to model:
| Position | DL360 Gen9 | DL560 Gen8 |
|---|---|---|
0 | 01-Inlet Ambient | 01-Inlet Ambient |
1 | 02-CPU 1 | 02-CPU 1 |
3 | 04-P1 DIMM 1-6 | 04-CPU 3 |
18 | 19-PS 1 Internal | 19-VR P1 |
19 | 20-PS 2 Internal | 20-VR P2 |
23 | 24-HD Controller | 24-VR P1 Mem1 |
A row reading position 18 as a power supply reads a voltage regulator on the DL560, and nothing says so: it is a plausible temperature under the wrong name. Reading by exact name does not cure it either, because the number is part of the name and differs too: the same array controller is 24-HD Controller on one and 32-HD Controller on the other. Two servers of the same model differ as well in which sensors are live, depending on the cards fitted.
So the temperature templates use the uf_ilotemps user function. It drops the number from each name and works out what each sensor is, so one template suits every model. It costs 4 extra credits a reading. Its answer replaces the document, so a sensor using it cannot also read the fans. Two sensors on the same document still share one login.
| Expression | Gives |
|---|---|
ambient | The inlet temperature: the room. |
exhaust | The air leaving the server, on models that have the sensor. |
cpu | The hottest processor. |
memory | The hottest memory sensor. |
raid | The array controller. |
drives | The hottest drive. |
network | The hottest network card that reports a temperature. |
pci | The hottest card in a PCI slot, whatever it is. |
psu1, psu2 | Each power supply, from PS 1 Internal or P/S 1, whichever the model has. |
hottest, hottest_name | The hottest sensor of all, and its name. |
headroom, tightest | The fewest degrees any sensor has below its own caution limit, and its name. |
over | How many sensors are at or past that limit. |
notok | How many report a health other than OK. |
flagged | The names of those, for the alarm. |
CPU 1, PCI 2, I/O Zone | Any other sensor, by its name without the number. |
A kind this server does not have is left out, so its slot stays empty and its rule is skipped. A sensor reporting itself absent is left out too, not given the 0 it reports.
The iLO Temps template
The short set, for charts. It reads /redfish/v1/Chassis/1/Thermal/ with no arguments:
| Slot | Reading | Expression | Chart |
|---|---|---|---|
SV03 | Ambient | ambient | 1 |
SV04, SV05 | CPU 1, CPU 2 | CPU 1, CPU 2 | 1 |
SV06, SV07 | PSU 1, PSU 2 | psu1, psu2 | 1 |
SV08, SV09 | CPU 3, CPU 4 | CPU 3, CPU 4, empty on a two processor server | 1 |
Its rules are the guard, status code not equal to 200 for CRIT, and Ambient greater than 30 for WARN and greater than 38 for CRIT. The inlet temperature is the one most worth an alarm, because it tells you about the room. The server's own limits for it were 42 and 46 on the servers we tested, so these give warning well before the server acts on it. The power supplies are charted without a rule, since iLO gives no limit for them to compare with: their limit fields read 0, meaning none is set.
The iLO Sea of Sensors template
The wider view, on the same document, so adding it to a device already running iLO Temps costs no further login. These go on Chart 2, so the two do not crowd one chart:
| Slot | Reading | Expression |
|---|---|---|
SV03 | Ambient | ambient |
SV04 | Exhaust | exhaust |
SV05 | CPU, hottest | cpu |
SV06 | Memory, hottest | memory |
SV07 | RAID card | raid |
SV08 | Drives, hottest | drives |
SV09 | Network cards | network |
SV10 | I/O zone | I/O Zone |
SV11 | PCI cards, hottest | pci |
SV12 | Hottest of all | hottest |
SV13 | Headroom | headroom |
SV14 | Over limit | over |
SV15 | Not OK | notok |
SD03 | Hottest sensor | hottest_name |
SD04 | Least headroom | tightest |
SD05 | Flagged | flagged |
| Rule as shipped | State | Catches |
|---|---|---|
Status code not equal to 200 | CRIT | The guard rule. |
Ambient greater than 30, then 38 | WARN, CRIT | The room. |
Over limit greater than 0 | CRIT | A sensor at or past the limit the server itself sets for it, whichever one it is. |
Not OK greater than 0 | WARN | A sensor the server reports as unhealthy. |
Headroom less than 5 | WARN | Anything within five degrees of its limit. These limits differ widely, 42 for the inlet and 100 for a controller, so this one rule does what would otherwise take one per sensor. |
Cards in PCI slots
A card in a slot reports as PCI 1, PCI 2 and so on, whatever it is. Only the server's built-in array controller reads as HD Controller, so an array controller on a card reads as nothing under raid until you name its slot in the user function's arguments:
{"raid": "PCI 2", "network": ["PCI 3", "PCI 4"]}
To see what is in each slot, read /redfish/v1/Systems/1/PCIDevices/ with Store raw response ticked, or look at the controller's System Information page. Each device there has a Name and a DeviceLocation such as PCI-E Slot 2.
Not every network card has a temperature to report. The built-in and FlexibleLOM 1Gb adapters on our DL360 Gen9s have sensors named LOM and LOM Card that report themselves absent, so network stays empty there. I/O Zone, the air around the cards, is the nearest thing, and every server we tested has it, which is why the template reads it as well.
Readings that mislead
- Processor temperatures can sit at exactly 40. On two of our servers both processors read 40 on every poll for a week. On a third they ranged from 40 up to 53, so 40 looks like a floor rather than a measurement. A flat line there is normal and says little about how busy the server is.
- Hot is normal for some parts. Array controllers and other cards read about 70 degrees on our healthy servers, with limits of 100, and
hottestwill usually be one of them. Judge byheadroomrather than by the highest figure. - Fans read as a percentage on iLO, of full speed, not in RPM. Our servers sat between 12 and 27 percent.
The iLO Power template
It reads /redfish/v1/Chassis/1/Power/, and needs no user function. These all go on Chart 3, watts on one scale and volts on the other:
| Slot | Reading | Expression |
|---|---|---|
SV03 | Power | PowerControl.0.PowerConsumedWatts |
SV04 | Average, 20 min | PowerControl.0.PowerMetrics.AverageConsumedWatts |
SV05 | Peak, 20 min | PowerControl.0.PowerMetrics.MaxConsumedWatts |
SV06, SV07 | PSU 1 and 2 output | PowerSupplies.0.LastPowerOutputWatts, then .1 |
SV08, SV09 | PSU 1 and 2 input | PowerSupplies.0.LineInputVoltage, then .1 |
SD03, SD04 | PSU 1 and 2 health | PowerSupplies.0.Status.Health, then .1 |
SD05, SD06 | PSU 1 and 2 state | PowerSupplies.0.Status.State, then .1 |
Its rules are the guard, each power supply's health not equal to OK for WARN, and each one's state not equal to Enabled for WARN. State is there because a part the server reports as absent has no health at all, so the health rule alone would skip a supply that has been pulled. Two supplies rarely share the load evenly: one of ours gave 5 watts and its partner 188, both healthy, so an uneven pair is no fault in itself. On a server with one supply the second supply's rows stay empty and their rules are skipped.
Other useful readings
| Reading | Document | Method and expression | Slot |
|---|---|---|---|
| How many power supplies | Power | JSONCOUNT PowerSupplies | Number |
| How many fans | Thermal, without a user function | JSONCOUNT Fans | Number |
| Fan speed | Thermal, without a user function | JSON Fans.0.CurrentReading | Number |
| Controller firmware | Manager | JSON Firmware.Current.VersionString | Text |
| iLO licence | Manager | JSON Oem.Hp.License.LicenseString | Text |
| Finished starting up | System | JSON Oem.Hp.PostState | Text |
| Entries in the hardware log | Systems/1/LogServices/IML/Entries/ | JSON Total | Number |
Several rows on one document cost one request, so add them to a sensor that already reads it rather than adding a sensor. A rule of changed since last time on the firmware row gives a note whenever somebody updates the controller.
Logs: the IML and syslog
iLO keeps two logs: the Integrated Management Log, or IML, for the hardware, and the iLO Event Log for the controller itself. A sensor cannot read their contents usefully. They list the oldest entry first, 30 to a page, and an extraction path can only point at one fixed place, so the newest entry cannot be reached.
iLO's remote syslog carries both logs, and it is the natural way out to the syslog sender. It is set under Administration > Management > Remote Syslog on iLO 4, and can depend on the controller's licence. Three things to know before relying on it:
- Match the controller's address, not its tag. The two logs arrive under different tags. On iLO 4 the controller's own events, such as
Server reset, are tagged#ILO4, while hardware entries from start-up, such as a cache module error, are taggedPOSTMessage, so a rule looking for#ILO4misses the ones that matter most. A rule naming the controllers insourceand nothing else ships all of it, whatever the tag:{"name": "ilo", "source": ["192.0.2.17", "192.0.2.19"]} - Syslog usually travels by UDP, with no second chance. Anything sent while the collector is down is gone. If the collector runs on the servers it watches, restarting them loses the very events a restart is likely to raise: we lost a storage battery failure this way.
- Leave out the logins. Each poll appears as a
Browser loginand often aBrowser logout. Six sensors polling each minute made nearly 14,000 lines a day from three servers, against 13 lines of anything else, so they bury what matters. An exclude rule, tested before the include above, drops them and nothing else, and a failed login still ships:{"name": "ilo-logins", "source": ["192.0.2.17", "192.0.2.19"], "match": "Browser log(in|out): "}
So treat syslog as the detail and the sensors as the safety net. A lost syslog line is not a lost fault: the health reading stays amber until the fault is cleared. And a reading of Total on Systems/1/LogServices/IML/Entries/ with a rule of up since last time by 1 gives notice of any new entry in the hardware log, whether or not its syslog line arrived. A repeat of an existing entry only updates its count, so does not raise the total.
The settings can be read from Managers/1/NetworkService/: Oem.Hp.RemoteSyslogEnabled, Oem.Hp.RemoteSyslogServer and Oem.Hp.RemoteSyslogPort. They are worth a look when a server's lines stop arriving. Avoid Chassis/1/Power/PowerMeter/, too: it carries a day of readings and came to 62 KB, close to the body cap, for nothing a rule would use.
Other generations, and Dell iDRAC
The pattern is the same throughout: a document, a path and a rule. What changes is the paths, and a path that does not resolve fails silently, so check every new generation with Store raw response before trusting a green sensor.
| Controller | What differs from iLO 4 |
|---|---|
| iLO 5 | Vendor fields sit under Oem.Hpe, not Oem.Hp. Recent firmware offers the standard Systems/1/Storage/ beside SmartStorage. |
| iLO 6 (Gen11) | SmartStorage has been removed: use Systems/1/Storage/. Thermal and Power remain, alongside newer documents. |
| iDRAC 7 and 8 | Redfish from firmware 2.40.40.40. |
| iDRAC 9 | Redfish from firmware 3.00.00.00, on every licence level. The object names are System.Embedded.1 and iDRAC.Embedded.1, as in the table above. |
| iDRAC 10 (17G) | From firmware 1.10.17.00. Check whether the Chassis document still links to Thermal and Power; if not, see below. |
On iDRAC
- The health sensor is the iLO Health template with the path
/redfish/v1/Systems/System.Embedded.1, whoseHealthRolluprows then apply. ReadStatus.HealthRollupas well asStatus.Health, since the first covers the parts beneath. - Power reads the same:
PowerControl.0.PowerConsumedWattsonChassis/System.Embedded.1/Power. - Temperatures read the same,
ReadingCelsiusin aTemperatureslist, but the names are Dell's own, such asSystem Board Inlet Temp, souf_ilotempsdoes not know what they are. Useuf_json_flattenwith the arguments{"key": "Name", "path": "Temperatures"}, and read each with an expression such asSystem Board Inlet Temp.ReadingCelsius. - Fans give
ReadingandReadingUnits, usually RPM, notCurrentReadingas a percentage. - Where a correct password gets a
401on iDRAC 9 firmware 7, look first at the controller's Redfish and authentication settings, since that firmware is reported to have changed how Basic authentication is offered.
The newer documents
The DMTF replaced Thermal and Power with ThermalSubsystem and PowerSubsystem in Redfish 2020.4, keeping the old ones for compatibility, and vendors are moving over at different speeds. Where the Chassis document no longer links to the old ones, look under the chassis for ThermalSubsystem/ThermalMetrics for the temperatures and PowerSubsystem for the power supplies. The system document's Status.Health has not moved, which is one more reason to start with the health sensor.