Dell Unity: Storage Processor Panic During A Pool Full Event

Storage Pool reached 100% full.

A storage process has panicked when a pool became full and one or more File Systems (FS) or Logical Unit Numbers (LUN) show as offline and requiring recovery. Any snapshots for one or more FS or LUN which are marked as offline need recovery as well.

The sequence of the panic:

1. The pool becomes 100% full, and the array starts invalidating the snapshots.
2. The array starts to unmount the snapshots.
3. The array tries to update the metadata and superblock in order to allocate getSlice. This fails due to the pool being 100% full.
4. The operation to unmount the snapshots timeout and cause an SP Panic.

The FS, LUNs, and snapshots need free space in the pool in order for a recovery attempt to be started. This is done by expanding the impacted pool.

If the impacted pool cannot be expanded, then free space must be made in the pool by one of the following methods:

• Delete snapshots
• Expand the storage pool (using existing drives or purchasing additional drives if there are slots available)
• Remove unneeded resources (LUNs or FSs)
• Run the space reclaim procedure (from the host-side) to potentially free-up space which has not been zeroed-out on the array side
  • Reclaiming VMFS deleted blocks on Thin Provisioned LUNs (external link)
• Investigate preallocated space concerns
  • Dell Unity: Pool Space Issues Due To Large Levels Of Preallocated Space

If further help is needed, contact Dell Technical Support and reference this Dell Knowledgebase Article.

How To Delete Unnecessary Snapshots 

Step 1: Check Pool Usage:

• Unisphere:
  • STORAGE > Pools
  • Select the affected pool and review the Used (%) column.
  • Click the pool name to view detailed usage, including space used by snapshots.
• CLI:
  • Log in to putty with service credentials.
  • Run command: uemcli -d 10.0.0.1 -u Local/joe -p MyPassword456! /stor/config/pool show -detail 

Step 2: Identify Large or Old Snapshots:

• Unisphere:
  • STORAGE > Block > LUNs or STORAGE > File > File Systems
  • For each LUN or file system, check the Snapshots tab.
• CLI: 
  • Log in to putty with service credentials.
  • Run command to see snapshot details: uemcli -d 10.0.0.1 -u Local/joe -p MyPassword456! /prot/snap show -detail 
  • Run command to see snapshot space used by individual storage resource: uemcli -d 10.0.0.1 -u Local/joe -p MyPassword456! /stor/config/pool/sr -pool <pool name> show -detail
  • Example:

uemcli -d 10.0.0.1 -u Local/joe -p MyPassword456! /stor/config/pool/sr -pool pool_1 show -detail 

 ID = sv_1
 Name = AF LUN 1
 Resource type = LUN
 Pool = pool_1
 Total pool space used = 14448566272 (13.4G)
 Total pool preallocated = 4610351104 (4.2G)
 Total pool snapshot space used = 4593991680 (4.2G)
 Total pool non-base space used = 4593991680 (4.2G)
 Health state = OK (5)
 Health details = "The LUN is operating normally. No action is required."

Step 3: Delete Snapshots:

⚠️ Caution: Ensure that the snapshots are not required for backup or recovery before deletion. ⚠️

• Unisphere:
  • Select one or more snapshots to delete.
  • Click Delete and confirm the action.
  • Repeat for other LUNs or FSs as needed.
• CLI:
  • Run command: uemcli -d 10.0.0.1 -u Local/joe -p MyPassword456! /prot/snap –id <Snapshot ID> delete

Step 4: Monitor Pool Usage:

• Unisphere:
  •  After deletion, return to STORAGE > Pools
  • Refresh the page and monitor the Used (%) value.
  • It may take a few minutes for the space to be reclaimed.

• CLI:
  • Run command: uemcli -d 10.0.0.1 -u Local/joe -p MyPassword456! /stor/config/pool show -detail 

Additional Tips:

• Automate snapshot deletion: Set retention policies to automatically delete old snapshots.
• Review snapshot schedules: Avoid overly frequent snapshot creation.
• Enable alerts: Configure alerts for pool usage thresholds (for example, 80%, 90%) to act proactively.

Related Documents: 

• Dell Unity™ Family Unisphere® Command Line Interface User Guide Version 5.5.1

Related Dell Knowledgebase Articles:

• Dell Unity: Automatic Snapshot Deletion Becomes Paused