PowerVault ME4: How to Troubleshoot CloudIQ Upload Failures

What is Dell AIOps "CloudIQ"?   

Dell rebranded CloudIQ as Dell AIOps. AIOps and CloudIQ may be used interchangeably. This article continues to use the original reference CloudIQ since PowerVault ME4 documentation and command-line interface continue to use this. 

• CloudIQ is a no-cost cloud-native application that proactively monitors and measures the overall health of your storage systems through intelligent, comprehensive, and predictive analytics.
• CloudIQ leverages the SaaS deployment model to add new features and functionality, pushing the upgrades to the cloud and exposing these new features to users regularly with no downtime.
• CloudIQ supports PowerEdge servers, PowerVault ME Series arrays, and more Dell infrastructure products. For more information, read the whitepaper Dell AIOps: A Detailed Review
• A Dell AIOps mobile app for iOS and Android which allows monitoring from anywhere
• Users must have a valid Dell customer business support account by registering using a proper corporate email address. For more, see How to Register for Access to Dell Technologies Online Support or Upgrade an Existing Account
• A paid (active) ProSupport or ProSupport Plus agreement and an accepted End User License Agreement (EULA) for each asset to be viewed within CloudIQ is required.
• PowerVault ME4 series controllers should be firmware G280 or later.
• SupportAssist must be enabled in PowerVault Manager or CLI to send data from the asset to CloudIQ hosted on Dell infrastructure.
• The service uses two-factor authentication, the CloudIQ/Dell AIOps user interface is https://aiops.dell.com/ Use Dell support username (email) and password and emailed response code to access.
• If you would like to use this free service to monitor your PowerVault ME array, follow the instructions outlined in Dell AIOps Observability: General Procedures to Onboard PowerVault ME Series Arrays

Verifying connectivity from the PowerVault ME4 array.

Open an SSH session with administrator-level credentials to the ME4 controller A management IP address. You can use a terminal application such as PuTTY.

The following commands can be used to examine the settings and show if the ME4 is sending data:   

show network-parameters
show dns-management-hostname
show dns-parameters
show support-assist
show cloud-iq 

What can be observed from this output?

• The management interfaces on both controllers are configured, enabled, and healthy. By default SupportAssist data is sent through controller A management interface unless controller A is offline.
• The hostname must be unique on each controller.
• Domain Name Server (DNS) IP addresses must be configured on each controller. Sometimes DNS settings may not get set after a controller is replaced causing SupportAssist to fail due to being unable to resolve the dell.com DNS names.
• SupportAssist must show enabled and whether your organization uses a direct TCP connection or a proxy server. SupportAssist establishes an outbound TCP connection on port 443 to any of the following addresses.
• CloudIQ must be enabled.

Stopping services on PowerVault ME4

If the settings are correct and no telemetry data is being sent, carry out the following commands. These commands are not production impacting and do not require a maintenance window.

You must wait a minute for the management controller (MC) services to complete restart.

set cloud-iq off
set support-assist off
restart mc both

If you are using SupportAssist with a proxy, ping your DNS servers and the proxy server to confirm these are accessible from the ME management interface on controller A (and B)

The Dell endpoint no longer responds to ICMP requests. Use Windows PowerShell on a local desktop to test if a TCP connection can be opened to the Dell endpoint on port 443 from your network. Linux users can use nc -zv stor.g3.ph.dell.com 443 to test if a TCP connection can be established on port 443. 

PS C:\WINDOWS\system32> Test-NetConnection stor.g3.ph.dell.com -Port 443 ComputerName : stor.g3.ph.dell.com RemoteAddress : 143.166.159.38 RemotePort : 443 InterfaceAlias : Ethernet 3 SourceAddress : 10.0.0.50 TcpTestSucceeded : True

Restarting service on PowerVault ME series

Once MC services have restarted and the ping response is verified, restart SupportAssist and CloudIQ services in the following order:   

set support-assist on
set cloud-iq on

If you do not onboard your PowerVault ME on CloudIQ, you can disable sending telemetry by turning off the service. To disable run the following command:  set cloud-iq off

Confirm connectivity.

Run the following commands to confirm connectivity and send a support bundle to Dell.

check support-assist
show support-assist
show cloud-iq
send support-assist-logs

When CloudIQ is enabled, the PowerVault SupportAssist service sends array performance I/O telemetry at 15 minute intervals and system configuration telemetry at 60 minute intervals. It may take an hour before you observe regular updates when you first onboard the PowerVault ME4. Log in to https://aiops.dell.com/ or use the Dell AIOps smartphone app to view your onboarded Dell infrastructure product status.

If the status of the PowerVault ME4 is still not being updated within CloudIQ, contact Dell technical support for further assistance. Include a support bundle if support assist logs are not being sent. For instructions, see Dell KB article: ME4: How to Collect support logs for ME4 models