EMC Storage Analytics: Sends false "LUN Down" alert for VNX Block Luns

EMC Storage Analytics generated "LUN down" Vrops alerts for all LUNs on a VNX Block array. The alerts were false and the LUNs were confirmed to have never went down.

This issue was caused when the ESA VNX block adapter was unable to execute Naviseccli commands to the VNX Storage Processor. When this occurs, the array is assumed to be down and alerts are created in vRealize. Upon the next successful data poll, the alerts will automatically clear in vRealize. This scenario can be caused by multiple issues, but the most likely causes are either a temporary network condition preventing communication between the ESA adapter and the VNX Storage Processor, or the VNX Storage Processor not being able to respond to the ESA adapter data poll. ESA adapter log errors that may indicate this issue might look similar to the following:
LOGICAL UNIT NUMBER 355
Name:  #######dev01_42_vdev01T2
UID:  60:06:01:60:02:70:36:00:48:F1:CD:18:14:D6:##:##
Current Owner:  SP A
Default Owner:  SP A
Allocation Owner:  SP A
UserA network error occurred while trying to connect: '##.###.###.###'.
Message :  Socket stream invalid
 
 
    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.lun.LunAdapterImp.getLuns(LunAdapterImp.java:317)
    at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.getLuns(VnxBlockDiscoveryService.java:309)
    at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.updateInfoFromArray(VnxBlockDiscoveryService.java:138)
    at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.discover(VnxBlockDiscoveryService.java:115)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.collect(VnxBlockCollector.java:364)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:505)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-01-05 17:02:36.502 [ERROR] [Collector worker thread 6] com.emc.pie.adapters.vnx.block.engine.ConnectionImp - Command Execution: Error returned from Block Storage System.
2017-01-05 17:02:36.504 [ERROR] [Collector worker thread 6] com.emc.vcops.collectors.vnxblock.VnxBlockCollector - connection test failed on SP ##.###.###.###
com.emc.pie.commons.exception.PlatformException:  
 
    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.initAgentInfo(AgentAdapterImp.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.getDeviceInfo(AgentAdapterImp.java:45)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.testSingleSPConnection(VnxBlockCollector.java:844)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.getSSLConnectionURL(VnxBlockCollector.java:794)
    at com.emc.vcops.moxy.EmcAdapterInstance.getConnectionURL(EmcAdapterInstance.java:1438)
    at com.emc.vcops.moxy.EmcAdapterInstance.isCertificateVerificationRequired(EmcAdapterInstance.java:566)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:439)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-01-05 17:02:46.615 [ERROR] [Collector worker thread 6] com.emc.pie.adapters.vnx.block.engine.ConnectionImp - Command Execution: Error returned from Block Storage System.
2017-01-05 17:02:46.615 [ERROR] [Collector worker thread 6] com.emc.vcops.collectors.vnxblock.VnxBlockCollector - connection test failed on SP ##.###.###.###
com.emc.pie.commons.exception.PlatformException:  
 
    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.initAgentInfo(AgentAdapterImp.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.getDeviceInfo(AgentAdapterImp.java:45)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.testSingleSPConnection(VnxBlockCollector.java:844)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.getSSLConnectionURL(VnxBlockCollector.java:794)
    at com.emc.vcops.moxy.EmcAdapterInstance.getConnectionURL(EmcAdapterInstance.java:1438)
    at com.integrien.alive.common.adapter3.AdapterBase.onCheckCertificate(AdapterBase.java:2277)
    at com.emc.vcops.moxy.EmcAdapterInstance.checkCertificate(EmcAdapterInstance.java:549)
    at com.emc.vcops.moxy.EmcAdapterInstance.isCertificateValid(EmcAdapterInstance.java:526)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:439)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-01-05 17:02:46.819 [ERROR] [Collector worker thread 6] com.emc.vcops.moxy.EmcAdapterInstance.1847 - check certificate failed.  reason Untrusted certificate found.
2017-01-05 17:42:36.508 [ERROR] [Collector worker thread 15] com.emc.pie.adapters.vnx.block.engine.ConnectionImp - Command Execution: Error returned from Block Storage System.
2017-01-05 17:42:36.509 [ERROR] [Collector worker thread 15] com.emc.vcops.collectors.vnxblock.VnxBlockCollector - connection test failed on SP ##.###.###.###
com.emc.pie.commons.exception.PlatformException:  
 
    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.initAgentInfo(AgentAdapterImp.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.getDeviceInfo(AgentAdapterImp.java:45)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.testSingleSPConnection(VnxBlockCollector.java:844)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.getSSLConnectionURL(VnxBlockCollector.java:794)
    at com.emc.vcops.moxy.EmcAdapterInstance.getConnectionURL(EmcAdapterInstance.java:1438)
    at com.emc.vcops.moxy.EmcAdapterInstance.isCertificateVerificationRequired(EmcAdapterInstance.java:566)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:439)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

There is no resolution to be performed from the EMC Storage Analytics product. This is an issue with either the customer network or the VNX Block array. Troubleshooting / resolution will have to take place from within the customer environment to include checking network logs to find any possible outages or degradation and viewing the logs on the VNX Storage Processors to try to identify any issues that may have prevented the VNX Storage Processors from responding to Naviseccli commands executed from the vRealize node.