Data Domain: DDVE Filesystem Down / Unable to Enable DDVE Filesystem

Summary: Unable to bring up DDVE Filesystem.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms



DDVE filesystem down.
Active tier disk is not available.
Active tier shows:

sysadmin@DD1# filesys show space 
Active Tier:
Resource                  Size GiB       Used GiB     Avail GiB    Use%   Cleanable GiB*
----------------          --------       --------     ---------    ----   --------------
/data: pre-comp                -              -           -          -                -
/data: post-comp               -              -           -          -                -
/ddvar                       49.2            10.0        36.7       21%               -
/ddvar/core                 158.5             7.0       143.5        5%               -
----------------          --------         --------  ---------     ----   --------------
The filesystem is unavailable at this time.

Cause

System Environment where DDVE is configured on ESX connected with external storage device.

kA23a00000009o8CAA_1_0

The above said issue is occurred when external storage device loses connectivity with ESXi host.
The following error reported in ESX server vmkernel log 

Log Location #cd /var/log/vmkernel.log

cpu4:2098003)ScsiDeviceIO: 3032: Cmd(0x45a2c0b7ca80) 0x2a, CmdSN 0x3bf from world 2305853 to dev "naa.6006016001f049003b705c5cde10cdfc" (vmhba5:C0:T5:L1) failed H:0x8 D:0x0 P:0x0
cpu8:2098003)ScsiDeviceIO: 3032: Cmd(0x459ac13b9340) 0x28, CmdSN 0x2e7 from world 2305853 to dev "naa.6006016001f0490045705c5c730bd671"  (vmhba5:C0:T5:L7) failed H:0x8 D:0x0 P:0x0
cpu7:2098003)ScsiDeviceIO: 3032: Cmd(0x45a2c0a2a380) 0x88, CmdSN 0x368 from world 2305853 to dev "naa.6006016001f049003d705c5ccf627f86"  (vmhba5:C0:T5:L3) failed H:0x8 D:0x0 P:0x0
cpu8:2098003)ScsiDeviceIO: 3032: Cmd(0x45a319d16c00) 0x8a, CmdSN 0x33e from world 2305853 to dev "naa.6006016001f049003f705c5c47140d37" (vmhba5:C0:T5:L5) failed H:0x8 D:0x0 P:0x0
cpu5:2098003)ScsiDeviceIO: 3047: Cmd(0x45a31708cb00) 0x2a, CmdSN 0x39a from world 2305853 to dev "naa.6006016007f04900c272965b8bff4a05" (vmhba5:C0:T5:L11) failed H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0

cpu0:66299)ScsiDeviceIO: 2954: Cmd(0x4397ba4ea9c0) 0x4d, CmdSN 0x88555a from world 613487900 to dev "naa.500003981809326d" failed H:0x0D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
cpu10:615767554)<6>usb usb2: resumed
cpu20:615767554)<6>usb 2-1: resumed
cpu1:66060)<6>usb 2-1: suspended
cpu3:66074)<6>usb usb2: suspended
cpu10:613606886 opID=9b42f169)World: 12235: VC opID esxcli-4e-1f1a maps to vmkernel opID 9b42f169
cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:channel=0, target=0, lun=0, action=0
cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:No media
cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:channel=0, target=1, lun=0, action=0
cpu10:613606886 opID=9b42f169)vmw_ahci[00000011]: scsiDiscover:No media
cpu10:613606886 opID=9b42f169)vmw_ahci[0000001f]: scsiDiscover:channel=0, target=0, lun=0, action=0
cpu10:613606886 opID=9b42f169)vmw_ahci[0000001f]: scsiDiscover:port initialization not complete
cpu10:613606886 opID=9b42f169)vmw_ahci[0000001f]: scsiDiscover:channel=0, target=1, lun=0, action=0
 

From the logs you can see the following devices have failed:

naa.6006016001f049003b705c5cde10cdfc
naa.6006016001f0490045705c5c730bd671
naa.6006016001f049003d705c5ccf627f86
naa.6006016001f049003f705c5c47140d37
naa.6006016007f04900c272965b8bff4a05
naa.500003981809326d

In addition to the above the following alerts will be posted:

cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L3 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L0 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L7 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L5 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L9 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L11 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L6 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L4 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L2 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L8 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L10 to APM00183214618 is dead.
cpu13:2097909)ALERT: PowerPath:Path vmhba5:C0:T5:L1 to APM00183214618 is dead.
 

In addition, there will be frame drop events on  vmhbaX  noted:

vmhba5: FRAME DROP event observed suggests a problem with Fibre Channel link/switch!.

Resolution

DDVE filesystem should get enabled after completing the following two tasks:
  1. Check the connection path (vmhbaX:C0:T5:LX) and re-establish the connection between switch and external storage device.
  2. After connection is established, Confirm no further failed devices are reported in "vmkernel.log".
  3. Reboot DDVE.

If the issue still exists after the above mentioned tasks are complete then collect the following and engage DELL EMC DD Support:

  1. Support bundle from DDVE.
  2. Generate Bundle from ESXi server.
  3. Logs from connecting switch.

Affected Products

Data Domain

Products

Data Domain, Data Domain Virtual Edition
Article Properties
Article Number: 000050471
Article Type: Solution
Last Modified: 30 May 2024
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.