Celerra/VNX: End users unable to access the CIFS shares due to AV servers not responding

SPE (Symantec Protection Engine) AV software was uninstalled from the AV servers causing the AV servers not to scan any of the shares from the VNX array.


Customers were not able to access the CIFS shares.


Lot of Antivirus blocked messages on SERVER_LOG revoking the share access from the end users.


2020-04-17 14:42:01: KERNEL: 3: 3: ThreadsServicesSupervisor: Service:ANTIVIRUS Pool:AV_Others BLOCKED for 407 seconds: Server operations may be impacted .
2020-04-17 14:57:01: KERNEL: 3: 3: ThreadsServicesSupervisor: Service:ANTIVIRUS Pool:AV_Others BLOCKED for 1307 seconds: Server operations may be impacted .
2020-04-17 15:12:01: KERNEL: 3: 3: ThreadsServicesSupervisor: Service:ANTIVIRUS Pool:AV_Others BLOCKED for 2207 seconds: Server operations may be impacted .
2020-04-17 15:27:01: KERNEL: 3: 3: ThreadsServicesSupervisor: Service:ANTIVIRUS Pool:AV_Others BLOCKED for 3107 seconds: Server operations may be impacted .
2020-04-17 15:42:01: KERNEL: 3: 3: ThreadsServicesSupervisor: Service:ANTIVIRUS Pool:AV_Others BLOCKED for 4008 seconds: Server operations may be impacted .
2020-04-17 15:57:01: KERNEL: 3: 3: ThreadsServicesSupervisor: Service:ANTIVIRUS Pool:AV_Others BLOCKED for 4908 seconds: Server operations may be impacted .
2020-04-17 16:12:01: KERNEL: 3: 3: ThreadsServicesSupervisor: Service:ANTIVIRUS Pool:AV_Others BLOCKED for 5808 seconds: Server operations may be impacted .

SERVER_LOG messages showing the AV server is having issues.

2020-04-17 14:14:09: VC: 3: 33: Error on CAVA server X.X.X.X: AV_NOT_FOUND.


Unable to run any SERVER_XXXX commands on the VNX box. 

[nasadmin@ ~]$ server_date server_2
server_2 :
Error 5: server_2 : Input/output errorTimed out

[nasadmin@ ~]$ server_viruschk server_2
server_2 :
Error 4020: server_2 : Input/output error
Timed out

The server_setup command also was not able to run to stop the viruschecking service on the Data mover which could have cleared the SMB blocked threads to resume the CIFS services.

[nasadmin@ ~]$ server_setup server_2 -P viruschk -o stop
server_2 :
Error 4020: server_2 : Input/output error
Timed out

Cause :- SPE (Symantec Protection Engine) was removed from the two of the AV severs which was scanning the NAS shares.The AV servers were responding to the ping commands but the actual scanning was not happening on these machines. This caused CIFS blocked threads revoking the share access.

The shutdown policy was set to "shutdown=Viruschecking" we were having the third AV server which was alive but it was overloaded with lot of scan requests causing the CIFS to block threads.

There are two possible ways to restore the services back :
1. We need to contact the Antivirus administrators to check the AV servers which are not scanning any of the shares and ask them to do a reboot of the AV server or kill the CAVA service on the AV server machines. Once they do it the resources will be freed up allowing the users to access the shares unprotected.

2. The next solution would be to do a force panic of the Data Mover by engaging VNX support and this will cause the entire Data mover to revoke the access of all the CIFS and NFS clients.
                                       

Most of the time AV servers would cause CIFS blocked threads not allowing the CIFS users to access the shares. We can restore the services back most of the time by stopping the viruschecking service on the Data mover. But there are rare case scenarios where AV servers cause issues due to AV software being removed or AV software services getting hung. In these conditions we need to involve the Security/Antivirus team to check the AV servers and do the triage. This will restore the services back as they would restart the AV servers or stop the AV agent on the AV servers which will release the resources and will enable the customer to access the shares.