Isilon: Gen6: Battery Test Failure, due to missing Node Firmware

Gen6 nodes may report BBU failures similar to the following:
 

Message:  Battery Test Failure:  Replace the battery backup unit in chassis JAC#########89 slot # as soon as possible.

Battery tests may fail for the following reasons:

1. The battery was replaced, and after replacement, the firmware shows as version 00.00.

2. The battery did not have a chance to fully charge before the automatic test was started.

Download the latest Node Firmware Package (NFP) and upgrade the firmware on the node.
 
The latest NFP can be downloaded from the link below:
https://www.dell.com/support/home/en-us/product-support/product/isilon/drivers

This can be performed internally, by a CE, or PS.
----------------------------------------------------------------
 
Once it is upgraded verified by running the following command.

# isi upgrade firmware devices|egrep "bcc|mongoose"

In OneFS 8.2 and above, to initiate a manual battery test run the following command on the affected node:

# isi_hwmon -s

In OneFS version 8.1.2.x, initiate a manual battery test by running the following commands directly on the affected node:

# isi services -a isi_hwmon disable
# mv /var/log/nvram.xml /var/log/nvram.xml.old
# isi services -a isi_hwmon enable

The test will start within 10 minutes of running the above commands and can be verified by running the following command:

# grep -i "battery test" /var/log/isi_hwmon.log

EXAMPLE Output:

2020-08-04T13:57:07-04:00 <3.5> RTANASISI04-15 isi_hwmon[14322]: Starting battery test
2020-08-04T14:08:57-04:00 <3.5> RTANASISI04-15 isi_hwmon[14322]: Battery test has completed
2020-08-04T14:08:57-04:00 <3.5> RTANASISI04-15 isi_hwmon[14322]: Battery Test Error Code: 0
2020-08-04T14:08:57-04:00 <3.5> RTANASISI04-15 isi_hwmon[14322]: Battery Test Error Mask: 0
2020-08-04T14:08:57-04:00 <3.5> RTANASISI04-15 isi_hwmon[14322]: Battery test passed
2020-08-04T14:08:57-04:00 <3.5> RTANASISI04-15 isi_hwmon[14322]: Next test is scheduled for: Sat Aug 15 00:08:57 2020
2020-08-04T14:08:57-04:00 <3.5> RTANASISI04-15 isi_hwmon[14322]: Clearing BIST results

DO NOT REBOOT THE NODE DURING THE TEST!
 
The test takes a minimum of 12 minutes to run, it can run for several hours.

The status and results can be verified with the same command:

# grep -i "battery test" /var/log/isi_hwmon.log

 The relevant results are "Battery test has completed/passed/failed." 

The test always reports:  Battery test has completed and another line after that reports Battery test passed or failed.

The Battery Test for Clusters running OneFS_v8.2.x.x and later versions has changed. 

EXAMPLE
 

IsilonGen6-M18-1# uname -r
v9.0.0.0
 
IsilonGen6-M18-1# isi_hw_status -i
SerNo: JACXXXXXXXXX96
Config: 110-365-400E-02
ChsSerN: JWXXXXXXXXXX70
ChsSlot: 1
FamCode: H
ChsCode: 4U
GenCode: 0
PrfCode: 4
Product: H400-4U-Single-64GB-1x1GE-2x10GE SFP+-30TB-800GB SSD
 
IsilonGen6-M18-1# isi_hwmon -s
-s,  --battery-test
Trigger a battery health test.  Tests take a long time so if an existing test is already in progress this action will not complete.
 
IsilonGen6-M18-1# tail -f /var/log/isi_hwmon.log
2020-08-21T15:34:13-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[15234]: Manual override detected.  Forcing battery test
2020-08-21T15:34:23-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[15273]: Starting battery test
2020-08-21T15:34:26-07:00 <3.6> IsilonGen6-M18-1 isi_hwmon[15288]: (isi_hwmon) INFO: --- Initializing isi_hwmon daemon ---
2020-08-21T15:34:26-07:00 <3.6> IsilonGen6-M18-1 isi_hwmon[15288]: (isi_hwmon) INFO: The block "isi.hw.hwmon.blocks.ecc_v2.ECC" is deprecated
2020-08-21T15:46:01-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[19623]: Battery test has completed
2020-08-21T15:46:01-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[19623]: Battery Test Error Code: 0
2020-08-21T15:46:01-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[19623]: Battery Test Error Mask: 0
2020-08-21T15:46:01-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[19623]: Battery test passed
2020-08-21T15:46:01-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[19623]: Next test is scheduled for: Tue Sep  1 01:46:01 2020
2020-08-21T15:46:01-07:00 <3.5> IsilonGen6-M18-1 isi_hwmon[19623]: Clearing BIST results

Remember to break out of the tail-f by pressing Ctrl+c.

IsilonGen6-M18-1# cat /etc/ifs/nvram.xml
<?xml version="1.0" encoding="utf-8"?>
<nvram>
  <battery_1>
    <test_result>Mon Aug 03 23:25:58 2020 : passed - battery tested passed [  ]</test_result>
    <test_result>Fri Aug 14 09:36:44 2020 : passed - battery tested passed [  ]</test_result>
    <test_result>Fri Aug 21 15:46:01 2020 : passed - battery tested passed [  ]</test_result>
  </battery_1>
</nvram>

Recommended Resources
Here are some recommended resources related to this topic that might be of interest:

• PowerScale: Isilon: Gen6: Addressing Generation 6 Battery Backup Unit (BBU) Test Failures - Event ID: 900160008
• Isilon: Battery replacement in nodes with battery tray