VNX: VNXe: A single disk array enclosure power supply fuse failure may lead to a disk array enclosure shutdown

Disk Array Enclosure (DAE) may shutdown due to a power supply firmware defect after a power supply fuse blows. This may result in data unavailability to the user.

This may occur on power supplies after approximately 3 years of run-time and if Operating Environment has not been recently upgraded.

This may impact all the VNX2, VNX1, and VNXe3300 arrays running older versions of Operating Environment and 5 drive 3U DAE with power supply part numbers 071-000-553 and 071-000-554.

 

There is a common disk enclosure software (CDES 1.55 version) available to fix this issue. This software is included in current Operating Environment releases.

• The preferred resolution is to upgrade the Block Operating Environment to 05.32.000.5.221 (or later), released on June 22, 2016. File and Unified systems will have a corresponding File OE upgrade.
• If an OE upgrade is not possible, select and download CDES 1.55 installation package from Dell Support
• This is applicable on all arrays running 05.32.xxx.x.xxx and 05.31.xxx.x.xxx. 
• It performs a rebootless upgrade of the CDES by copying the CDES to the correct location on the array, then kicking off the upgrade process.
• After NDU reports the operation is completed successfully, the user needs to wait for sometime to confirm that CDES has been updated. This is because, from NDU operation point of view, it is completed, but the CDES upgrade operation is a background task and will take a while to complete (the more enclosures, the longer it takes).
• If an additional enclosure is added with an older version of CDES, then CDES 1.55 package should be installed again to upgrade the CDES. 

If upgrading the Operating Environment, use Unisphere Service Manager (USM) and follow normal processes

1. Instructions are available from mydocuments.emc.com > VNX Series > Update VNX software

If installing the CDES 1.55 package on a VNX1 Block array follow these steps:

NOTE:    A rapid response procedure has been developed to bring an enclosure on line if the customer is currently experiencing a DU condition due to enclosure shutdown while waiting for power supply replacement. This procedure is applicable to R32 and R33 systems only. Open a Surge escalation with Engineering for recovery if power supply replacement will take some time.
 
 
Once the recovery is completed with EE assistance, send e-mail to Charlie Stoneham and let him know that L2/RT TSE with assistance from Engineering (name of the engineering that assisted) successfully recovered CDES issue using the procedure


   How To Target Upgrade CDES Firmware For Inyo 
 

1. Log in to the array using Unisphere Service Manager (USM).
2. Select Software > System Software > Prepare for Installation (Step-1)
3. Follow the Step-1 wizard. Browse and select the CDES 1.55 .ndu file you downloaded earlier and hit Next until you finish all the steps
4. Select Software > System Software > Install Software (Step-2) in the USM, click Next, and follow the steps in the Step-2 wizard until the CDES 1.55 software is loaded on the array.
5. Once CDES 1.55 is loaded on the array, you can verify by clicking LCC and Power Supply Firmware Status in the bottom right Tools section of USM. Check show details in the LCC firmware tab and you can see the current LCC firmware version 1.55.

If installing the CDES 1.55 package on a VNX1 File or Unified system follow these steps:

1. Log in to the array using Unisphere Service Manager (USM).
2. Select Software > System Software > Prepare for Installation (Step-1)
3. Select the third option, Verify storage environment for VNX OE for Block and/ or Enabler installation , click Next.
4. Follow the wizard. Browse and select the CDES 1.55 .ndu file you downloaded earlier and hit Next until you finish all the steps
5. Select Software > System Software > Install Software (Step-2) in the USM, click Next.
6. Select the third option, Install VNX OE for Block and/ or Enabler, Next, and follow the steps in the wizard until the CDES 1.55 software is loaded on the array
7. Once CDES 1.55 is loaded on the array, you can verify by clicking LCC and Power Supply Firmware Status in the bottom right Tools section of USM. Check show details in the LCC firmware tab and you can see the current LCC firmware version 1.55.

For VNXe3300 arrays:

• Select and download CDES 1.55 installation package from Dell Support
• This is applicable on all arrays running v2.4.0.20932 and later only. Note: If you try to download CDES 1.55 on systems running software versions earlier than v2.4.0.20932 it will fail with an error message in the GUI: "Image not compatible".
• It performs CDES upgrade on reboot (one SP at a time). It copies the CDES to the correct location on the array, then reboots the SP, after reboot CDES upgrade happens automatically.
• After NDU reports the operation is completed successfully, the user needs to wait for sometime to confirm that CDES has been updated. This is because, from NDU operation point of view, it is completed, but the CDES upgrade operation is a background task and will take a while to complete (the more enclosures, the longer it takes).
• If an additional enclosure is added with an older version of CDES, then CDES 1.55 package should be installed again to upgrade the CDES.  

For VNXe3300 arrays follow these steps:

1. Log in to the array using Unisphere.
2. Select Settings > More configuration > Update Software
3. Upload Candidate Version > Browse and select the CDES 1.55 .gpg file you downloaded earlier and select Upload.
4. Once uploaded, Install the candidate version and wait for the installation to complete.
5. Once CDES 1.55 is loaded on the array, you can verify by checking the SAS expander version. The SAS expander version will change to 0155. You can see that in the Unisphere GUI. VNXe > System > System Health and click on the DAE LCC.

VNX2 customers must upgrade to R33.9.184 OE release (or later). 
VNX1 customers recommended to upgrade to the VNX OE R32.221 release that was made available on June 22, 2016.

Update, May 10, 2017 
A system with the affected part numbers may already have CDES 1.55 firmware version installed that addresses this issue. If this is the case, no further action is needed. The CDES version can be validated as follows:

VNX1 and VNX2:

1. Launch Unisphere Service Manager (USM).
2. Log in to the system. The Block OE version is displayed on the right side of the main page.
3. Alternately, below the OE version under Tools, click on LCC and Power Supply Firmware Status, then View Details. Each DAE6S should have a Version of LCC of 1.55 or later.

VNXe:

1. Launch Unisphere.
2. Navigate to VNXe > System > System Health.
3. Click on DAE LCC. The SAS expander version should be 0155 or later.

For VNX1 and VNX2 arrays:

1. Login to the array via Unisphere
2. System-->Hardware-->Storage Hardware-->Disk Array Enclosures
3. Expand Disk Array Enclosures in the Unisphere GUI
4. For each Disk Array Enclosure select and expand Power Supplies in the Unisphere GUI
5. Select each Power Supply and select "View Resume" at the bottom in the Unisphere GUI.
6. View Resume will bring up the power supply details in a pop-up window. If the power supply part numbers are 071-000-553 or 071-000-554 then the array is impacted by this issue and the user needs to load the CDES 1.55 software on the array.

For VNXe3300 arrays:

1. Login to the VNXe3300 array via Unisphere
2. VNXe-->System-->System health
3. Select and expand each Disk Array Enclosure in the Unisphere GUI
4. Select the Disk Array Enclosure Power Supply.
5. Once the power supply is selected you will see the part number of the power supply on the right side in the Unisphere GUI. If the power supply part number is 071-000-553 or 071-000-554 then the VNXe3300 array is impacted by this issue and the user needs to load the CDES 1.55 software on the array.