Start a Conversation

Unsolved

This post is more than 5 years old

T

5443

January 19th, 2016 11:00

ESA 3.4 and VNX Storage Processor Objects Missing

As the title suggests, after upgrading from 3.3 to 3.4 all of our VNX2 Storage Processor objects are missing.  I've tried to run both a "New Objects Only" discovery as well as a complete discovery for the EMC adapters but it does not find any storage processor objects.  Everything else seems to be working as expected (disks, LUNs, Storage Pools, etc).

Unfortunately for us, the limiting factor on our 7600s are the CPUs which means that the dashboards don't show any information.  Below is a screenshot from the default VNX Overview dashboard CPU widget.  (Note: There are 2 types of "Storage Processor" object types but both are blank) Thoughts?

CPU Widget.png

January 19th, 2016 13:00

We also have the SCOM Management Pack which does not automatically clean-up objects removed in SCOM.  Unfortunately during a round of SCOM purges, any object whose collection status wasn't green was removed.

The VNX Topology dashboard does not list any storage processors.  Rather, in order from top to bottom, it has:

vCenter Server -> Datacenter -> Host System (physical ESXi hosts) -> Virtual Machine -> Datastore -> EMC Adapter Instance -> VNX Physical Host (these are the host/initiator objects) -> LUN -> Tier -> FAST Cache -> Storage Pool -> RAID Group -> Disk -> SP Front End Port

As for now, all of the EMC adapters are a collection status of "Data receiving"

13 Posts

January 19th, 2016 13:00

If the Storage Processor resources are not being collected post-upgrade then they should still exist, but their resource status would be marked as down and they would appear as red with a health score of zero. 

Do you see any Storage Processor resources in the VNX Topology dashboard or EMC Storage Topologies Inventory Tree?

Also, if you navigate to Home > Administration > Solutions and select the EMC Adapter, is the VNX Block adapter instance in question showing a collecting state with data receiving?

13 Posts

January 19th, 2016 13:00

Okay that makes sense, the Storage Processor resources were most likely deleted when all down resources were purged.  In any case, we now need to determine why the Storage Processors are not being collected.  Would you check the VNX Block adapter instance logs for any error messages, and if any exist report them here?

January 19th, 2016 14:00

I went ahead and re-installed the pak file for 3.4 (not doing a full reset) for the heck of it.  I ran a discovery of new objects only but it didn't return anything.

I assume the adapter logs are the files in /data/log/vcops/log/adapters/EmcAdapter?  If so, one array has 2 logs, one has 3, and one has ~10 along with a default EmcAdapter.log

Which should I be looking for?

13 Posts

January 19th, 2016 15:00

Look for any log entries tagged [ERROR] or containing exception in log files in the the adapter instance log folders under the one you mentioned.

ie /data/log/vcops/log/adapters/EmcAdapter/ - / - .log

January 20th, 2016 06:00

This exists for at least a couple of arrays.

2016-01-20 08:52:32.391 [DEBUG] [Task Processor worker thread 6] com.emc.vcops.moxy.EmcAdapterInstance.15013 - constructed adapter=adapterId[id='15013',name=' ']

2016-01-20 08:52:42.095 [ERROR] [Collector worker thread 7] com.emc.vcops.moxy.EmcAdapterInstance.15013 - unexpected collect exception

java.lang.NullPointerException

        at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.updateInfoFromArray(VnxBlockDiscoveryService.java:137)

        at com.emc.vcops.collectors.vnxblock.VnxBlockDiscoveryService.discover(VnxBlockDiscoveryService.java:112)

        at com.emc.vcops.collectors.vnxblock.VnxBlockCollector.collect(VnxBlockCollector.java:364)

        at com.emc.vcops.moxy.EmcAdapterInstance.collect(EmcAdapterInstance.java:490)

        at com.emc.vcops.moxy.EmcAdapterInstance.onCollect(EmcAdapterInstance.java:381)

        at com.integrien.alive.common.adapter3.AdapterBase.collectBase(AdapterBase.java:679)

        at com.integrien.alive.common.adapter3.AdapterBase.collect(AdapterBase.java:490)

        at com.emc.vcops.EmcAdapter.collect(EmcAdapter.java:505)

        at com.integrien.alive.collector.CollectorWorkItem3.run(CollectorWorkItem3.java:44)

        at com.integrien.alive.common.util.ThreadPool$WorkerItem.run(ThreadPool.java:253)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

13 Posts

January 20th, 2016 11:00

That looks helpful, thanks.  I'm looking into this to determine why you're encountering this error.  I attempted to reproduce the issue that you're seeing in my environment but so far I've been unsuccessful.  I deployed ESA 3.3, monitored a VNX Block array for a few hours, confirmed that Storage Processors were being collected, then upgraded to ESA 3.4.  After the upgrade I am still getting Storage Processor data in ESA, dashboards, etc...  Can you let me know which VNX OE software revision you are using?  That may help to reproduce the issue.  Thanks.

January 20th, 2016 11:00

We're running 05.33.008.5.119 on all of the arrays.

13 Posts

January 20th, 2016 20:00

Check whether both SPs are online for your VNX systems encountering this issue.

Also, it's recommended to file a support request and submit a full log bundle so we can root cause and resolve the exception.

https://support.emc.com/servicecenter/contactEMC/

No Events found!

Top