thomsonac1
1 Copper

ESA 3.4 and VNX Storage Processor Objects Missing

As the title suggests, after upgrading from 3.3 to 3.4 all of our VNX2 Storage Processor objects are missing.  I've tried to run both a "New Objects Only" discovery as well as a complete discovery for the EMC adapters but it does not find any storage processor objects.  Everything else seems to be working as expected (disks, LUNs, Storage Pools, etc).

Unfortunately for us, the limiting factor on our 7600s are the CPUs which means that the dashboards don't show any information.  Below is a screenshot from the default VNX Overview dashboard CPU widget.  (Note: There are 2 types of "Storage Processor" object types but both are blank) Thoughts?

CPU Widget.png

Tags (3)
0 Kudos
9 Replies
Highlighted
JoshHutt
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

If the Storage Processor resources are not being collected post-upgrade then they should still exist, but their resource status would be marked as down and they would appear as red with a health score of zero. 

Do you see any Storage Processor resources in the VNX Topology dashboard or EMC Storage Topologies Inventory Tree?

Also, if you navigate to Home > Administration > Solutions and select the EMC Adapter, is the VNX Block adapter instance in question showing a collecting state with data receiving?

0 Kudos
thomsonac1
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

We also have the SCOM Management Pack which does not automatically clean-up objects removed in SCOM.  Unfortunately during a round of SCOM purges, any object whose collection status wasn't green was removed.

The VNX Topology dashboard does not list any storage processors.  Rather, in order from top to bottom, it has:

vCenter Server -> Datacenter -> Host System (physical ESXi hosts) -> Virtual Machine -> Datastore -> EMC Adapter Instance -> VNX Physical Host (these are the host/initiator objects) -> LUN -> Tier -> FAST Cache -> Storage Pool -> RAID Group -> Disk -> SP Front End Port

As for now, all of the EMC adapters are a collection status of "Data receiving"

0 Kudos
JoshHutt
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

Okay that makes sense, the Storage Processor resources were most likely deleted when all down resources were purged.  In any case, we now need to determine why the Storage Processors are not being collected.  Would you check the VNX Block adapter instance logs for any error messages, and if any exist report them here?

0 Kudos
thomsonac1
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

I went ahead and re-installed the pak file for 3.4 (not doing a full reset) for the heck of it.  I ran a discovery of new objects only but it didn't return anything.

I assume the adapter logs are the files in /data/log/vcops/log/adapters/EmcAdapter?  If so, one array has 2 logs, one has 3, and one has ~10 along with a default EmcAdapter.log

Which should I be looking for?

0 Kudos
JoshHutt
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

Look for any log entries tagged [ERROR] or containing exception in log files in the the adapter instance log folders under the one you mentioned.

ie /data/log/vcops/log/adapters/EmcAdapter/<Adapter_Instance_Name>-<ID>/<Adapter_Instance_Name>-<ID>.log

0 Kudos
thomsonac1
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

This exists for at least a couple of arrays.

2016-01-20 08:52:32.391 [DEBUG] [Task Processor worker thread 6] com.emc.vcops.moxy.EmcAdapterInstance.15013 - constructed adapter=adapterId[id='15013',name='<ARRAY NAME>']

2016-01-20 08:52:42.095 [ERROR] [Collector worker thread 7] com.emc.vcops.moxy.EmcAdapterInstance.15013 - unexpected collect exception

java.lang.NullPointerException

        at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.updateInfoFromArray(VnxBlockDiscoveryService.java:137)

        at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.discover(VnxBlockDiscoveryService.java:112)

        at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.collect(VnxBlockCollector.java:364)

        at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:490)

        at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:381)

        at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:679)

        at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:490)

        at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)

        at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:44)

        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

0 Kudos
JoshHutt
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

That looks helpful, thanks.  I'm looking into this to determine why you're encountering this error.  I attempted to reproduce the issue that you're seeing in my environment but so far I've been unsuccessful.  I deployed ESA 3.3, monitored a VNX Block array for a few hours, confirmed that Storage Processors were being collected, then upgraded to ESA 3.4.  After the upgrade I am still getting Storage Processor data in ESA, dashboards, etc...  Can you let me know which VNX OE software revision you are using?  That may help to reproduce the issue.  Thanks.

0 Kudos
thomsonac1
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

We're running 05.33.008.5.119 on all of the arrays.

0 Kudos
JoshHutt
1 Copper

Re: ESA 3.4 and VNX Storage Processor Objects Missing

Check whether both SPs are online for your VNX systems encountering this issue.

Also, it's recommended to file a support request and submit a full log bundle so we can root cause and resolve the exception.

https://support.emc.com/servicecenter/contactEMC/

0 Kudos