Storage Analytics: Sends False "LUN Down" Alert for VNX Block LUNs

Storage Analytics generated "LUN down" Vrops alerts for all LUNs on a VNX Block array. The alerts were false, and the LUNs were confirmed to have never gone down.

This issue was caused when the ESA VNX block adapter was unable to run "Naviseccli" commands to the VNX Storage Processor. When this occurs, the array is assumed to be down and alerts are created in vRealize. Upon the next successful data poll, the alerts automatically clear in vRealize. Multiple issues can cause this scenario. But the most likely causes are either a temporary network condition preventing communication between the ESA adapter and the VNX Storage Processor, or the VNX Storage Processor not being able to respond to the ESA adapter data poll. ESA adapter log errors that may indicate that this issue might look similar to the following:

LOGICAL UNIT NUMBER 355
Name:  #######dev01_42_vdev01T2
UID:  60:06:01:60:02:70:36:00:48:F1:CD:18:14:D6:##:##
Current Owner:  SP A
Default Owner:  SP A
Allocation Owner:  SP A
UserA network error occurred while trying to connect: '##.###.###.###'.
Message :  Socket stream invalid


    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.lun.LunAdapterImp.getLuns(LunAdapterImp.java:317)
    at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.getLuns(VnxBlockDiscoveryService.java:309)
    at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.updateInfoFromArray(VnxBlockDiscoveryService.java:138)
    at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.discover(VnxBlockDiscoveryService.java:115)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.collect(VnxBlockCollector.java:364)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:505)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-01-05 17:02:36.502 [ERROR] [Collector worker thread 6] com.emc.pie.adapters.vnx.block.engine.ConnectionImp - Command Execution: Error returned from Block Storage System.
2017-01-05 17:02:36.504 [ERROR] [Collector worker thread 6] com.emc.vcops.collectors.vnxblock.VnxBlockCollector - connection test failed on SP ##.###.###.###
com.emc.pie.commons.exception.PlatformException:

    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.initAgentInfo(AgentAdapterImp.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.getDeviceInfo(AgentAdapterImp.java:45)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.testSingleSPConnection(VnxBlockCollector.java:844)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.getSSLConnectionURL(VnxBlockCollector.java:794)
    at com.emc.vcops.moxy.EmcAdapterInstance.getConnectionURL(EmcAdapterInstance.java:1438)
    at com.emc.vcops.moxy.EmcAdapterInstance.isCertificateVerificationRequired(EmcAdapterInstance.java:566)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:439)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-01-05 17:02:46.615 [ERROR] [Collector worker thread 6] com.emc.pie.adapters.vnx.block.engine.ConnectionImp - Command Execution: Error returned from Block Storage System.
2017-01-05 17:02:46.615 [ERROR] [Collector worker thread 6] com.emc.vcops.collectors.vnxblock.VnxBlockCollector - connection test failed on SP ##.###.###.###
com.emc.pie.commons.exception.PlatformException:

    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.initAgentInfo(AgentAdapterImp.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.getDeviceInfo(AgentAdapterImp.java:45)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.testSingleSPConnection(VnxBlockCollector.java:844)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.getSSLConnectionURL(VnxBlockCollector.java:794)
    at com.emc.vcops.moxy.EmcAdapterInstance.getConnectionURL(EmcAdapterInstance.java:1438)
    at com.integrien.alive.common.adapter3.AdapterBase.onCheckCertificate(AdapterBase.java:2277)
    at com.emc.vcops.moxy.EmcAdapterInstance.checkCertificate(EmcAdapterInstance.java:549)
    at com.emc.vcops.moxy.EmcAdapterInstance.isCertificateValid(EmcAdapterInstance.java:526)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:439)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2017-01-05 17:02:46.819 [ERROR] [Collector worker thread 6] com.emc.vcops.moxy.EmcAdapterInstance.1847 - check certificate failed.  reason Untrusted certificate found.
2017-01-05 17:42:36.508 [ERROR] [Collector worker thread 15] com.emc.pie.adapters.vnx.block.engine.ConnectionImp - Command Execution: Error returned from Block Storage System.
2017-01-05 17:42:36.509 [ERROR] [Collector worker thread 15] com.emc.vcops.collectors.vnxblock.VnxBlockCollector - connection test failed on SP ##.###.###.###
com.emc.pie.commons.exception.PlatformException:

    at com.emc.pie.adapters.vnx.block.engine.Parser.checkActionResponseForErrors(Parser.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.initAgentInfo(AgentAdapterImp.java:62)
    at com.emc.pie.adapters.vnx.block.engine.agent.AgentAdapterImp.getDeviceInfo(AgentAdapterImp.java:45)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.testSingleSPConnection(VnxBlockCollector.java:844)
    at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.getSSLConnectionURL(VnxBlockCollector.java:794)
    at com.emc.vcops.moxy.EmcAdapterInstance.getConnectionURL(EmcAdapterInstance.java:1438)
    at com.emc.vcops.moxy.EmcAdapterInstance.isCertificateVerificationRequired(EmcAdapterInstance.java:566)
    at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:439)
    at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:403)
    at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:674)
    at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:487)
    at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)
    at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:46)
    at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

There is no resolution to be performed from the Storage Analytics product. This is an issue with either the customer network or the VNX Block array. Troubleshooting/resolution have to take place from within the customer environment to include checking network logs to find any possible outages or degradation and viewing the logs on the VNX Storage Processors to try to identify any issues that may have prevented the VNX Storage Processors from responding to "Naviseccli" commands ran from the vRealize node.