PowerVault ME4, ME5: Investigating Hosts Losing Access to Array
Summary: This article is intended for storage administrators who use Dell PowerVault ME4/ME5 arrays. It explains how to collect the right evidence, perform first‑pass checks, and identify common causes when hosts lose access to mapped volumes. Where deeper host operating system or switch configuration is required, consult the documentation from the operating system or switch vendor after completing the array‑side validation steps. ...
Instructions
Purpose and scope
This article is intended for storage administrators who use Dell PowerVault ME4/ME5 arrays. It explains how to collect the right evidence, perform first‑pass checks, and identify common causes when hosts lose access to mapped volumes. Where deeper host operating system or switch configuration is required, consult the documentation from the operating system or switch vendor after completing the array‑side validation steps.
Connectivity overview
- Serial Attached SCSI (SAS) is supported as direct‑attached storage (DAS) only. SAS switches are not supported with ME arrays. Use dual paths to both controllers for redundancy.
- Internet Small Computer Systems Interface (iSCSI) and Fibre Channel (FC) support direct attached and switch attached designs. Use two fabrics or subnets for resiliency.
- Validate the host drivers, firmware, and transceivers before making any changes.
1. Collect the right evidence first.
Gather this information before making changes. Correlate everything to the incident timeframe and time zone.
From the array (ME4/ME5)
- Generate the support bundle in PowerVault Manager by selecting Maintenance, then Support, then Collect Logs. The bundle includes controller events, firmware levels, and frontend port status.
- Review Alerts and Events in PowerVault Manager. Record any controller reboot, failover, link reset, or enclosure event that aligns with the incident.
What to look for: link down states, speed or negotiation mismatches, rising error counters, or missing initiators compared to the expected design.
From Windows Server hosts
Check MPIO and confirm that devices are claimed and that the expected number of Active/Optimized paths is present. Export relevant Disk, StorPort, and Multipath IO(MPIO) events from Event Viewer around the incident. Compare HBA (host bus adapter) and network card driver and firmware versions to the latest versions available.
From VMware ESXi hosts
Search the vmkernel.log for APD (All Paths Down), PDL (Permanent Device Loss), and path transition events.
Confirm that each datastore device shows the expected number of paths and that the policy is appropriate for ME arrays, such as Round Robin.
esxcli storage core path list
esxcli storage core device list
From Linux hosts
Confirm that multipath shows multiple active paths per WWID and correlate kernel transport errors to the incident time. Validate HBA driver and firmware versions.dmesg -T | egrep -i 'scsi|sas|reset|error'
multipath -ll
From the network or fabric (if used)
- iSCSI switches: Capture per‑port CRC or drop counters, link flaps, MTU (maximum transmission unit) settings, and flow control policy. Ensure that MTU is consistent end to end when using jumbo frames, and enable at least receive flow control on storage ports. Use MPIO for redundancy instead of LACP on host iSCSI NICs.
- FC fabrics: Collect fabric logs for FLOGI/PLOGI, RSCNs, and per‑port error counters on both fabrics. Validate that the zoning matches the intended design.
2. Triage checks that resolve most cases.
-
Confirm the intended topology. If the controller uses SAS, ensure a direct‑attach design without a SAS switch and maintain dual paths to both controllers. For iSCSI and FC, confirm a two‑fabric or two‑subnet layout.
-
Review the storage events and port health. Align event timestamps with host errors. Verify that frontend ports are up at the expected speed and that initiators are logged in. Investigate any port with rising error counters.
-
Validate host multipathing.
- Windows Server with Multipath I/O (MPIO): Devices should be claimed and show the intended number of Active/Optimized paths.
- VMware ESXi: Devices should show a full path count and an appropriate policy, such as Round Robin.
- Linux with device‑mapper multipath: Each mpath dev should show multiple active paths.
- Check transport stability.
- iSCSI: two isolated subnets and VLANs (virtual local area networks), consistent MTU, and receive or symmetric flow control. Replace link aggregation with MPIO for data paths.
- FC: two fabrics and consistent zoning; look for link resets and login failures.
- SAS: dual‑path cabling to both controllers and no switch in the path; check HBA tools for PHY errors and reseat or replace suspect cables.
- Confirm software and firmware versions. Compare host HBA and network card drivers and firmware to the Support Matrix and update if required.
3. Transport‑specific guidance
SAS (Direct-attached)
- Use a dual‑path DAS design to both controllers. Keep within supported cable lengths. Do not place a SAS switch in the data path.
- For Windows Server clustering with SAS HBAs, follow operating system guidance if a device is identified as RAID rather than an HBA.
- Refer to the Deployment Guide for cabling diagrams and examples.
iSCSI (Direct-attached or Switch-attached)
- Use two dedicated iSCSI subnets and VLANs with consistent MTU, and enable receive or symmetric flow control. In direct‑attach designs, connect one NIC per subnet to a port on the corresponding controller and use MPIO.
- For VMware ESXi, use the software iSCSI initiator. iSCSI offload cards are not supported with ME5. Use one VMkernel per fabric on a vSwitch with a single vmnic.
Fibre Channel (DAS or SAN)
- Use two fabrics. A common practice is single‑initiator, single‑target zoning: Use one host HBA World Wide Port Name (WWPN) with one array port WWPN per zone and repeated on both fabrics.
4. Common causes and how to confirm them.
- Controller failover or firmware issue: Array events show a controller reboot or failover that aligns with host APD or PDL in logs. Address firmware if release notes indicate relevant fixes.
- Multipathing misconfiguration: Hosts show a reduced path count or report dead paths. Correct MPIO, NMP, or multipath settings to restore expected pathing.
- iSCSI network issues: Switch counters show drops or CRC errors, or MTU is inconsistent. Enable receive or symmetric flow control, align MTU, and replace link aggregation with MPIO.
- FC zoning or fabric instability: Affected hosts fail to log in on one fabric or show frequent link resets; correct zoning or remediate optics and cabling.
- SAS cable or port degradation: Rising PHY errors or intermittent link down states are visible in array or HBA diagnostics; reseat or replace cables.
- Unsupported drivers or firmware: Only certain hosts or HBAs are affected; updating to Support Matrix versions resolves the condition.
5. Escalation checklist
Provide the following items when opening a service request:
- The array Support Bundle and the precise incident window with time zone
- A topology diagram of the host connections and fabrics and subnets, including which hosts lost access and which paths were affected.
- Host evidence:
- Windows Server: Check MPIO and relevant Event Viewer entries.
- VMware ESXi: Excerpts from vmkernel.log and the output of esxcli storage commands
- Linux: Dmesg excerpts and multipath output.
- Switch or fabric evidence: iSCSI counters and configuration snippets, and FC fabric logs and zoning screenshots.
- Version inventory: Array firmware (in the bundle) and host HBA or NIC driver and firmware versions.
6. Dell documentation to consult can be found on support.dell.com under the specific device model
- Support Matrix: Supported operating systems, HBAs, NICs, and rules
- Deployment Guides: Cabling, initial setup, and topology examples.
- Administrator’s Guide: This is for day‑to‑day management.
- CLI Reference: Complete list of show commands for investigation.