Dell Unity: Performance charts are unavailable because the /EMC/backend/metricsluna1 partition is full.

UI performance charts are not available.

When logging into the SSH console as service, you can see that the metricsluna1 partition is full:

13:18:52 service@FNMXXXXX spa:~/user# df -h
Filesystem         Size  Used Avail Use% Mounted on
udev               1.2G  8.0K  1.2G   1% /dev
/dev/mapper/eroot   13G  6.3G  5.9G  52% /
tmpfs              4.2G   91M  4.2G   3% /dev/shm
tmpfs              1.7G  180M  1.6G  11% /run
tmpfs              4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/sda1          107M  122K  101M   1% /mnt/c4lx-cfg_msata
/dev/sda7          127G  1.2G  120G   1% /cores
/dev/ram0           57M  2.5M   54M   5% /tmp
/dev/sda9          2.0G  341M  1.7G  17% /var/lib/docker
/pramfs            122M   41M   81M  34% /pramfs
/dev/mirrora5       57G  388M   54G   1% /EMC/backend/service
/dev/c4nasdba1     995M   54M  892M   6% /nbsnas
/dev/c4nasdba2     3.5G  248M  3.1G   8% /EMC/backend/CEM
/dev/c4loga2       4.6G   92K  4.4G   1% /EMC/backend/perf_stats
/dev/c4loga1       3.3G   69M  3.1G   3% /EMC/backend/log_shared
/dev/metricsluna1   16G  16G   0  100% /EMC/backend/metricsluna1
/dev/loop0         192M   12M  169M   7% /EMC/backend/CEM/ese/hostmnt
/dev/c4fastvpa1    6.0G  812K  5.7G   1% /EMC/fastvp

Listing the file usage in this partition confirms it is filled with archived files:

13:20:35 service@FNMXXXXX spa:~/user# du -h /EMC/backend/metricsluna1/ |sort -hr |head -20
du: cannot read directory '/EMC/backend/metricsluna1/lost+found': Permission denied
du: cannot read directory '/EMC/backend/metricsluna1/metricsdb': Permission denied
15G    /EMC/backend/metricsluna1/archives
16G    /EMC/backend/metricsluna1/
16K     /EMC/backend/metricsluna1/lost+found
4.0K    /EMC/backend/metricsluna1/metricsdb

This is due to a code issue that causes old performance archive files to not be deleted at the backend. It occurs after the Operating Environment (OE) is upgraded to version 5.1, if a Unity array enables CloudIQ. This issue eventually leads to the metricsluna1 partition to fill up.

If the Unity array meets the following two criteria, then it is experiencing this code issue:

• Unity array runs on UnityOS 5.1.0 or 5.1.1
• CloudIQ is enabled

Resolution: The fix is available in UnityOS 5.1.2 and newer; the array must be upgraded. 

Workaround: Manually delete the old archive files using the tar method from KB 191159: Dell EMC Unity: Upgrade to 5.1 code may fail due to lack of inodes or Metrics Database inconsistency (Dell EMC Correctable).

When inode usage is full, the guidance in the above referenced KB article must be referred to and followed.

This file can be saved for the customer to use for archive metrics if necessary.

To free up available space, issue the following commands as root:

• Change dir to home directory:
  • root@FNMXXXXX spa:~# cd /home/service/user

• Send archive files to a tarball in the home directory:
  • root@FNMXXXXX spa:~/home/service/user# tar cvfz metrics_archive.tgz /EMC/backend/metricsluna1/archives/

• Confirm that the tarball file is valid:
  • root@FNMXXXXX spa:~/home/service/user# tar -tvf metrics_archive.tgz

• Delete the archive files to free up space:
  • root@FNMXXXXX spa:~/home/service/user#  rm /EMC/backend/metricsluna1/archives/_default*

• Verify that enough space is available:
  • root@FNMXXXXX spa:~/home/service/user# df -h