PowerScale: Smartfail may Not Complete if the Peer Node has a Full var Partition

When a node is being smartfailed and its peer node has a full /var partition, the FlexProtect job may appear to make little to no progress. The job may appear stuck in phase 6, device removal.

For example, if node 3 is being smartfailed and node 4 has a full /var partition, the FlexProtect job may not complete as nodes 3 and 4 are peer nodes.

A full /var partition can prevent normal cluster services and activities from functioning correctly. 

In this example, the /var partition is at 109% capacity. 

Filesystem               512-blocks          Used         Avail Capacity     iused        ifree %iused  Mounted on
/dev/mirror/root1           4045904       2275960       1446272    61%       42702       252720   14%   /
devfs                             2             2             0   100%           0            0  100%   /dev
/dev/mirror/var1            4062544       4061056       -323512   109%        1971       260171    1%   /var
/dev/mirror/var-crash       6087472        244896       5355584     4%          32       406174    0%   /var/crash
/dev/mirror/keystore         126544            56        116368     0%           3         8187    0%   /keystore
/dev/md0                     122332          4572        107976     4%         238        16400    1%   /tmp/ufp
/dev/md1.uzip                857646        799866        -10830   101%        2652          418   86%   /base
/dev/md2                       7132          2196          4368    33%         543          735   42%   /tmp/ipmi
/dev/md3                     122332            16        112532     0%           4        16634    0%   /hwmon
OneFS                 3019651793280 1465967775616 1455912400544    50% 93263665626 109978539654   46%   /ifs

Clean up the /var partition on the peer node using the various methods detailed in the below articles:

• Isilon: Event notification: The var partition is near capacity, Event ID: 100010001
• Isilon The /var/crash partition of a node in the cluster has reached 90% capacity alert - Event ID: 100010002 
• OneFS: Event notification: Node reached 95% or greater used capacity on the root file system (Log in as a registered Dell Support user is required to view this article.)