Isilon : LWSM service list not responding

Missing base.uzip can also cause issues starting other services like webui services and httpd services, which will affect GUI access on affected node.

• When LWSM service has issues it will give following errors like below when we use command like following :-
  
  /usr/likewise/bin/lwsm list
Error: ERROR_FILE_NOT_FOUND (2)

• While checking JWT status:

/usr/likewise/bin/lwsm status jwt

LW_ERROR_STRING_CONV_FAILED
Failed to convert string format (wide/ansi)

• Or when we try to start service by using command below 
  
  /usr/likewise/sbin/lwsmd --syslog --start-as-daemon
Error: ERROR_GEN_FAILURE (31)
   

Isilon-1# isi auth ads list
		Unable to connect to authentication daemon: ERROR_FILE_NOT_FOUND

Isilon-1# /usr/likewise/bin/lwio list
		zsh: no such file or directory: /usr/likewise/bin/lwio

When we list /base.uzip file we will see on one of the node /base file is missing 

Isilon-1# isi_for_array -s 'ls -lh /base.uzip'
Isilon-1: ls: /base.uzip: No such file or directory
Isilon-2: -rwxr-xr-x 1 root  wheel    27M Aug  2  2018 /base.uzip
Isilon-3: -rwxr-xr-x 1 root  wheel    27M Aug  2  2018 /base.uzip

This could occur due to root partition gets full which can show similar output like below if we run df -h command :-

Isilon-1# df -h
Filesystem               Size    Used   Avail Capacity  Mounted on
/dev/mirror/root0        1.9G    1.8G   -126M   107%    /
devfs                    1.0K    1.0K      0B   100%    /dev
/dev/mirror/var0         956M    147M    732M    17%    /var
/dev/mirror/var-crash    2.8G    1.7M    2.6G     0%    /var/crash
/dev/mirror/keystore      60M    4.0K     55M     0%    /keystore
/dev/md0                  60M    1.2M     54M     2%    /tmp/ufp
/dev/md1                 3.5M    422K    2.8M    13%    /tmp/ipmi
OneFS                    130T    3.7T    119T     3%    /ifs

 So please make sure there is no other data being written to the root partition, only /ifs should be utilized for customer data.

Please check which node has /base.uzip file missing :

Isilon-1# isi_for_array -s 'ls -lh /base.uzip'
Isilon-1: ls: /base.uzip: No such file or directory
Isilon-2: -rwxr-xr-x 1 root  wheel    27M Aug  2  2018 /base.uzip
Isilon-3: -rwxr-xr-x 1 root  wheel    27M Aug  2  2018 /base.uzip

Please copy the base.uzip file from good node which has ubase.uzip file to bad node, then reboot the node.

Example:

1. Copy base.uzip from existing node to a global dir like Isilon_Support:

isi_for_array -n2 'cp /base.uzip /ifs/data/Isilon_Support/'

2. copy base.uzip from backup dir to an affected node:

 isi_for_array -n1 'cp /ifs/data/Isilon_Support/base.uzip /'

3. Verify md5 hash value matches other nodes and check permissions:

isi_for_array -s 'ls -lh /base.uzip'

isi_for_array -s 'md5 /base.uzip'

NOTE: if permissions do not match and need to be adjusted, use command below on affected node:

chmod 755 /base.uzip

• verify permissions match:

isi_for_array -s 'ls -lh /base.uzip'