VPLEX: SymptomCode 0x8a4830dc, UPS load power lower than expected

A lower than expected power issue was reported for the UPS indicated in the CDATA of the call home message as the sample below illustrates.

Only Dual-engine and Quad-engine clusters contain Uninterruptible Power Supplies (UPS).
On the VS2, the UPS-A is for battery power backup of the Fibre-Channel switch-A and the management server, and the UPS-B for the Fibre-Channel switch-B.
On the VS6, the UPS-A and B are for the Infini-Band (IB) switches only.

Sample of the dial home that is sent for this issue:
VS2:

SymptomCode : 0x8a4830dc
Category : Status
Severity : Warning
Status : Warning
Component : DIRECTOR
ComponentID : director-1-2-B << component reporting the issue
SubComponent : ZPEM
CallHome : Yes
FirstTime : 20xx-05-07T01:45:58.048Z
LastTime : 20xx-05-07T01:45:58.048Z
Count : x
CDATA[ups@UPS-B: PartNo 078-000-052/SerialNo QS1303140800 /RevNo 1XE1 :UPS load power is lower than expected [Versions:MS{x.x.x.x}, Director{x.x.x.x.x}]
RCA: The fibre channel switch and/or management server are not plugged in to the UPS.

In this sample dial home output the issue reports that the UPS load power is lower than expected on UPS-B of cluster-1.

The cause of these dial homes being reported for lower than expected load power can be as follows:

VS2:
For UPS-A
The management server and/or the internal rack-mounted Fibre-Channel Switch A are not seen by the UPS.

For UPS-B
For the Internal rack mounted Fibre-Channel Switch B is not seen by the UPS.

VS6:
For UPS-A
The internal rack-mounted Infini-Band (IB) Switch A one or both power connections are not seen by the UPS.

For UPS-B
For the Internal rack mounted Infini-Band (IB) Switch B one or both power connections are not seen by the UPS.

Permanent Fix:
Dell VPLEX Engineering is currently investigating this problem. Once a fix is available, this article will be updated.

The purpose of this article is to verify the state of whichever UPS that is being reported for "lower than expected power levels."

To understand how to determine which UPS is being reported for this issue and from which cluster look at the ComponentID in the call home message shown in the Issue section above. In the call home message example, it shows "director-1-2-B" reported the issue. The first value noted is "1," this is the cluster-id so in this example, cluster-1. Because the report came from the "B" director, "director-1-2-B," this tells you it is UPS-B that the issue was reported for. Also, the CDATA tells you which UPS the call home report was for, "ups@UPS-B." The "UPS" are always connected to engine 2 on any cluster.

Director-1-2-B
               \  \  \------------ This tells you it is UPS-B (UPS-A's are attached to the "A" directors)
                 \  \-------------- This says it is the director is in engine 2.
                   \--------------- This tells you it is cluster-1.

To check if this issue is real or a false-positive:

VS2:

1. Verify that both fibre channel switch power cables are plugged into the appropriate UPS and that the UPS' are plugged in to each Power Distribution Unit (PDU) located on the mounting rails of the rack.
2. If the event is reported for UPS-A, verify the management server and FC COM SWITCH A (above the management server) power cables are plugged into UPS-A.
3. If the event is reported for UPS-B, verify the FC COM SWITCH B's power cable is plugged into UPS-B.
4. Check for loose power cables from the FC COM switches and management server.
5. Check that the serial cables from each UPS are securely attached.
6. Reference the appropriate configuration serial and power cabling diagram attached to this article.

VS6:

1. Verify that both Infini-Band (IB) switch power cables are plugged in to the appropriate UPS and that the UPSs are plugged in to each PDS Unit.
2. If the event is reported for UPS-A, verify the Infini-Band switch A power cables are plugged into UPS-A.
3. If the event is reported for UPS-B, verify the Infini-Band switch B power cables are plugged into UPS-B.
4. Check for loose power cables from the IB switches.
5. Check that the serial cables from each UPS are securely attached.
6. Reference the appropriate configuration serial and power cabling diagram attached to this article.

To check if the UPS came back to a healthy-state, in the examples below cluster-1 will be used:

1. Login into the management server for the VPLEX cluster the call home was reported from

   Sample output:

   login as: service
   Using keyboard-interactive authentication.
   Password:
   Last login: Day Month Date HH:MM:SS Year from x.x.x.x < IP Address the login is from
   service@ManagementServer:~>

2. Next access the VPlexcli using the same credentials used to access the management server

   Sample output on a VPLEX VS2 running pre-6.x, for 6.x versions the second login to access the VPlexcli is not required:

   service@ ManagementServer:~> vplexcli
   Trying ::1...
   Connected to localhost.
   Escape character is '^]'.
   
   Enter User Name: service
   
   Password:
   Creating logfile:/var/log/VPlex/cli/session.log_service_localhost_Logfile_T24531_YYYYMMDDHHMMSS
   
   VPlexcli:/>

3. Change Directory (cd) to the UPS context and list out the ups'

   Sample output:

   VPlexcli:/> cd /clusters/cluster-1/uninterruptible-power-supplies
   VPlexcli:/clusters/cluster-1/uninterruptible-power-supplies> ll
   Name
   -------
   ups-2-a
   ups-2-b

4. Browse to the UPS context level that the call home alert was for, and list out the details. For this example if you look back at the sample call home message in the Issue section and look at the CDATA. You see ups@UPS-B, and if you look above this at the "ComponentID" you see "director-1-2-B." The first number indicates the cluster, in this case it is "1" indicating cluster-1. Go to cluster-1, and then drill down to the "uniterruptible-power-supplies" context then look at ups-2-b as shown below.

   Sample output:

   VPlexcli:/clusters/cluster-1/uninterruptible-power-supplies> cd ups-2-b
   VPlexcli:/clusters/cluster-1/uninterruptible-power-supplies/ups-2-b> ll

   VS2:

   Name                      Value
   ------------------------  -------------
   battery-replacement-date  03/23/10 < date battery was installed in the UPS, if more than 3 years old the UPS must be replaced
   battery-status            fully-charged  << check this value
   battery-time-remaining    5min
   operational-status        online         << check this value
   part-number               078-000-052    << note the part number of the UPS
   revision-number           1XE1
   serial-number             <UPS Serial Number>

   VS6:

   Name                      Value
   ------------------------  -------------
   battery-replacement-date  09/21/15
   battery-status            fully-charged  << check this value
   battery-time-remaining    5min
   operational-status        online         << check this value
   part-number               078-000-079    << note the part number of the UPS
   revision-number           FFF
   serial-number             <UPS Serial Number>

   Check for a False-Positive event:

5. If the "battery-status" shows as fully-charged and the "operational-status" shows as online, the issue can be ignored. The alert was most likely a false positive. The issue may have been reported during a battery test/charging cycle being performed by the UPS. This can be confirmed by checking the firmware logs with the date closet to the date of the call home, which can be noted from the call home message sample listed in the Issue section, look for 'LastTime' date and then doing a search on the UPS 'serial number' listed in the CDATA of the call home in the firmware log.

   Example of battery test/charging cycle of UPS as reported in firmware logs:

   VS2:
   The report of the load power lower than expected:

   128.221.253.38/xmmg/log:5988:W/"154559":109:<4>20xx/11/04 10:36:26.54: ZPEM/220 ups@UPS-B: PartNo 078-000-052 /SerialNo QS1322142106 /RevNo 1XE1 : UPS load power is lower than expected.

   The report of the ups being faulted:

   128.221.253.38/xmmg/log:5988:W/"154559":110:<3>20xx/11/04 10:36:26.54: ZPEM/471 ups@UPS-B: PartNo 078-000-052 /SerialNo QS1322142106 /RevNo 1XE1 : The operational state of the specified fru is Faulted.

   The report of the ups battery as charging:

   128.221.252.38/xmmg/log:5988:W/"154559":111:<4>20xx/11/04 10:36:38.40: ZPEM/211 ups@UPS-B: PartNo 078-000-052 /SerialNo QS1322142106 /RevNo 1XE1 : battery-status is Charging <<< this tells us the UPS is in battery test or charging mode

   The report of the ups now working, no longer seen as faulted:

   128.221.252.70/xmmg/log:5988:W/"154559":112:<6>20xx/11/04 10:36:38.40: ZPEM/87 ups@UPS-B: PartNo 078-000-052 /SerialNo QS1322142106 /RevNo 1XE1 : The operational state of the specified fru has changed to Working. << this says the battery test/charging phase is over and the UPS is back on AC

   Check if UPS is really faulted:

6. If the "battery-status" shows charging and the "operational-status" shows online, check again in about 5 minutes as the battery test/charging mode may still be in-progress. If after five minutes the "battery-status" shows charging and the "operational-status" shows offline or failure, this may indicate an issue with the UPS, and you must have the faulted UPS replaced.

7. For the UPS replacement, you must contact VPLEX Support and let them know your findings and that the UPS must be replaced. Mention this article.

How to contact Dell VPLEX Support using Chat:
To reach support start by going to Dell Support.com. When at the Welcome to Dell Support page, at the top where you see "Search Dell or Identify your product" enter VPLEX Series, VPLEX VS2 or VPLEX VS6, then scroll down and look on the right side and look for "Contact Technical Support." If you have an Active Service Contract, sign in. If you do not have an Active Service Contract, reach out to your local Dell Representative for further assistance. With the Active service contract, and after you have signed in you should see the Technical Issues page and here you see the options on how to contact VPLEX Support.