Data Domain: Storage enclosure numbering is not sequential

Summary: This article describes scenarios where the shelf numbers, assigned to external storage enclosures (ES-30 & DS-60) on Data Domain appliances may not be sequential. Non-sequential numbering or gaps in the shelf numbering sequence may occur after a completed storage migration operation, chassis replacement for one of the storage enclosures, or physically moving a storage enclosure to a new path. Data Domain recommends that the steps detailed in this article be applied in order to re-number all of the enclosures sequentially. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Future enhancements to Data Domain Operating System code will be made to mitigate the excessive rescan and discovery operations when shelf numbering is non-sequential. Until such changes are made , it is recommended that the necessary steps are applied to ensure that the shelf number is sequential after a physical change or serial number change which causes a gap in sequential shelf numbering.


Beginning in DDOS 5.7.1, storage shelf numbering is tied to the storage shelf serial number. We call this a shelf "persistent-id".
A storage enclosure can be listed as "Offline" under multiple different scenarios where a physical change has been made to the SAS topology.
In this example, a "storage migration" was recently completed, and the source storage enclosures were physically removed. Due to each storage enclosure having an assigned "persistent-id", there are now several "Offline" shelves (2,3,4,8) that are no longer attached, These "offline shelves are no longer required for the Filesystem to be intact and operational. The fact that there are phantom shelves or non-sequential numbering of attached storage enclosures, does not affect the overall operation of the Data Domain Filesystem, however it can cause excessive rescan and discovery operations during boot-up as well as SNMP cores when a query is made to a phantom enclosure. This results in longer than expected boot times and the Volume not being available immediately after the boot phase is complete.

Use the #enclosure show summary command to list all enclosures which have ever been issued a persisten-id along with the current state of the enclosure.

#Enclosure Show Summary


----------------------
Enclosure   Model No.   Serial No.       State     OEM Name   OEM Value   Capacity
---------   ---------   --------------   -------   --------   ---------   --------
1           DD9800      NVT10164000024   Online                           4 Slots
2           DS60        APM00164936021   Offline                          60 Slots
3           DS60        APM00164802045   Offline                          60 Slots
4           DS60        APM00164802044   Offline                          60 Slots
5           DS60        APM00164902721   Online                           60 Slots
6           DS60        APM00164936019   Online                           60 Slots
7           DS60        APM00164936020   Online                           60 Slots
8           ES30        APM00164964966   Offline                          15 Slots
9           DS60        APM00164902719   Online                           60 Slots
10          DS60        APM00164902720   Online                           60 Slots
11          DS60        APM00164902722   Online                           60 Slots
12          ES30        APM00164964967   Online                           15 Slots
---------   ---------   --------------   -------   --------   ---------   --------
 
 
Use the #enclosure show topology CLI command to list which storage enclosures are currently attached.
Looking at the SAS Topology confirms "Offline" shelves (2,3,4,8) are not really attached, and that the numerical number assigned to the attached shelves is non-sequential.


#Enclosure Show Topology
-----------------------
Port       enc.ctrl.port       enc.ctrl.port
----   -   -------------   -   -------------
2a     >    9.B.H: 9.B.E   >   10.B.H:10.B.E
2b     >    5.B.H: 5.B.E   >   11.B.H:11.B.E
2c     >    7.B.H: 7.B.E
2d     >   12.B.H:12.B.E
3a     >   10.A.H:10.A.E   >    9.A.H: 9.A.E
3b     >    6.B.H: 6.B.E
3c     >    7.A.H: 7.A.E
3d     >   12.A.H:12.A.E
6a     >   11.A.H:11.A.E   >    5.A.H: 5.A.E
6b     >    6.A.H: 6.A.
6d
----   -   -------------   -   -------------
Encl   WWN                Serial #
----   ----------------   --------------
5      N/A                APM00164902721
6      N/A                APM00164936019
7      N/A                APM00164936020
9      N/A                APM00164902719
10     N/A                APM00164902720
11     N/A                APM00164902722
12     N/A                APM00164964967
----   ----------------   --------------

 
Note: Various outputs in the GUI or Autosupport may show the "Offline" shelf as absent

 8.1   (absent)
 8.2   (absent)
 8.3   (absent)
 8.4   (absent)
 8.5   (absent)
 8.6   (absent)
 8.7   (absent)
 8.8   (absent)
 8.9   (absent)
 8.10  (absent)
 8.11  (absent)
 8.12  (absent)
 8.13  (absent)
 8.14  (absent)
 8.15  (absent)

SNMP queries to phantom enclosures results in errors and subsequent cores. Corresponding messaging will be posted to the messages.engineering log


 # snmpwalk -v2c -c Public localhost -t 5 -r 5 dataDomainMib
Timeout: No Response from localhost
Apr 20 01:01:54 xx snmpd: WARNING: ddr/sm/snmp/enclosurePack_access.c:161      **** Enclosure id 3 is offline.
Apr 20 01:01:54 xx ddr_procmon: NOTICE: MSG-PMON-00008: Failure on /usr/sbin/snmpd.
Apr 20 01:01:54 xx ddr_procmon: WARNING: MSG-PMON-00011: Restarting /usr/sbin/snmpd.
Apr 20 01:01:56 xx ddfs[11215]: NOTICE: ddr_key_del_temp: deleting key-v2-100
Apr 20 01:02:04 xx snmpd: WARNING: ddr/sm/snmp/enclosurePack_access.c:161      **** Enclosure id 3 is offline.
Apr 20 01:02:04 xx ddr_procmon: NOTICE: MSG-PMON-00008: Failure on /usr/sbin/snmpd.
Apr 20 01:02:04 xx ddr_procmon: WARNING: MSG-PMON-00011: Restarting /usr/sbin/snmpd.
Apr 20 01:02:14 xx snmpd: WARNING: ddr/sm/snmp/enclosurePack_access.c:161      **** Enclosure id 3 is offline.
Apr 20 01:02:14 xx ddr_procmon: NOTICE: MSG-PMON-00008: Failure on /usr/sbin/snmpd.


Apr 20 14:02:42 xx snmpd: WARNING: dd_netsnmp_set_request_error: snmp error 129 occured
 
 

Cause

Scenarios which can trigger this condition are as follows:
  • Physically moving storage enclosures from one Data Domain appliance to another Data Domain appliance
  • Chassis replacement on any in-use storage enclosure (new chassis will introduce a  new serial number)
  • Storage Migration - after a migration is completed, the source shelves are no longer needed by the FS, and will be physically removed.
  • #storage remove CLI command was issues.

Resolution

To force the shelf "persistent-id" to re-number in sequential order, employ the following commands in SE mode. 
           

SE# enclosure show persistent-id   
The output will detail information from enclosures that are attached , and shelves that are no longer attached, along with their corresponding persistent-id. you will see that there are serial numbers which are no longer attached, or have been removed from the SAS topology. The ones I have highlighted are no longer attached, and #enclosure show summary will list them as "Offline".


Serial No. Model No. Persistent ID
-------------- --------- -------------
APM00164936021  DS60 2      <<< Offline
APM00164802045  DS60 3      <<< Offline
APM00164802044  DS60 4      <<< Offline
APM00164902721  DS60 5      
APM00164936019  DS60 6      
APM00164936020  DS60 7      
APM00164964966  ES30 8      <<< Offline
APM00164902719  DS60 9      
APM00164902720  DS60 10     
APM00164902722  DS60 11     
APM00164964967  ES30 12   
-------------- --------- -------------
12 enclosure(s) persisted.
 
Use the #enclosure release persistent-id CLI to release the existing persistent-id's.  You can release only the ones that are offline, or you can choose to release <all> of them.  On the next boot-up, each attached shelf will be issued a new persistent-id and there will be no gaps in shelf numbering.

NOTE: On HA Systems, it is recommended to run "enclosure release persistent-id all" on both nodes, one at a time beginning with the active node before issuing system reboot. Rebooting the active node will trigger a failover to the standby node. Once, the HA reboot cycle is complete and system is highly available again, reboot the current active node to complete the process. 



SE# enclosure release persistent-id
              {<serialno>|<persistent-id>|all}
    
 
Example:                                            
 #enclosure release persistent-id APMxxxxxxxxxxxxxx
 #enclosure release persistent-id 12
 #enclosure release persistent-id all

Next, reboot the system to allow new persistent ID's to be issued to each storage enclosure.


#system reboot


After the system has rebooted ,verify that the shelf numbering is sequential and there are no storage enclosures listed as "Offline".


# enclosure show summary
Enclosure   Model No.   Serial No.       State    OEM Name   OEM Value   Capacity
---------   ---------   --------------   ------   --------   ---------   --------
1           DD9800      NVT10162700023   Online                          4 Slots 
2           DS60        APM00164802045   Online              0           60 Slots
3           DS60        APM00164802043   Online              0           60 Slots
4           DS60        APM00164802044   Online              0           60 Slots
5           DS60        APM00164851510   Online              0           60 Slots
6           DS60        APM00164936021   Online              0           60 Slots
7           DS60        APM00164802046   Online              0           60 Slots
8           ES30        APM00164964966   Online                          15 Slots
---------   ---------   --------------   ------   --------   ---------   --------
8 enclosures present. 


 
SE# enclosure show persistent-id
Serial No.       Model No.   Persistent ID
--------------   ---------   -------------
APM00164802045   DS60        2            
APM00164802043   DS60        3            
APM00164802044   DS60        4            
APM00164851510   DS60        5            
APM00164936021   DS60        6            
APM00164802046   DS60        7            
APM00164964966   ES30        8            
--------------   ---------   -------------
7 enclosure(s) persisted.
 

Affected Products

Data Domain

Products

Data Domain, DD OS, DD OS Licensed Features
Article Properties
Article Number: 000054913
Article Type: Solution
Last Modified: 19 Jul 2023
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.