PowerStore: How to generate and collect various logs from PowerStore

Support Materials

Support Materials or Data Collects (DCs) are the most common logs for analysis and troubleshooting. There are various methods for running and collecting Support Materials. Use the method that is more convenient depending on the system state and the kind of access that is available.

Notes:

• If the issue being investigated is related to PowerStore replication or metro volumes, logs from all clusters and appliances should be collected and provided to Technical Support.
• If the PowerStore is not yet initialized (no management IPs yet), see PowerStore: How to connect to and collect support materials from an un-configured system.

Using the PowerStore Manager user interface

1. Log in to the PowerStore Manager user interface (UI) with your username and password.
2. Click the Settings menu or the system name.
3. Go to Gather Support Materials under the Support section.
4. Press the GATHER SUPPORT MATERIALS button. A new submenu appears on the right side.
5. The default options should be:
   • Description: Optional
   • Collection Timeframe (v3+ only): Leave is set to default (All) unless instructed to select something else by Technical Support.
   • Object Type: Appliances
   • Objects to collect data for: Select all appliances.
   • Send materials to Support when finished: Optional (requires SupportAssist).
   • Advanced support materials collection options: Do not select unless support asks for it. Note: Selecting the advanced options greatly increases the size of the Support Materials. Only chose this option when instructed to do so by your support representative.
6. Press START. This process may take up to 60 minutes depending on the size of the system.
7. Monitor the progress in the Jobs section of PowerStore Manager or within the Support Materials section.
8. Once the process above has finished, select the bundle to download from the Support Materials Library under the Gather Support Materials section.
9. Press DOWNLOAD or SEND TO SUPPORT
10. A submenu is displayed on the right side with a download link for each of the appliances.
11. Click Support material collection for all the appliances.

Using ssh/cli

Before you begin:

• IMPORTANT: Do not attempt to connect a USB device to the USB port on the PowerStore node as this may lead to unpredictable system behavior. For further details, see PowerStore: Universal Serial Bus (USB) Port Use Guidelines.
• Running and downloading logs from CLI/ssh should not be required unless the PowerStore Manager user interface is not accessible.
• Review the article PowerStore: Understanding shell limitations due rbash and other security implementations
• Each appliance only contains its own logs. So if it is a multi-appliance cluster, connect with WinSCP to each appliance separately and download the respective logs. The details about each appliance IP address can be found in the PowerStore Manager user interface under Settings > Network IPs.

How to:

1. Log in to the system over ssh with the service user and password.
2. The service command to generate Support Materials is svc_dc.
   For a multi-appliance system, include all appliances in the collection, example for a two appliance cluster:

   svc_dc run -a A1,A2

   
   For detailed usage, run svc_dc -h or svc_dc run -h, or see the PowerStore Service Scripts Guide available from the Dell Support pages.
3. When the logs finish, download them.
   In order to facilitate this process, links to the logs are automatically created in the home directory (/home/service/user) when the logs finish running. List them with ls -l. If for some reason the link is not created, list the files manually, this is how that can be done:
   
   List the available Support Materials (DCs) with the service command svc_dc list. Example:

   [SVC:service@xxxxxxx-A user]$ svc_dc list start_timestamp      id             status   dump     description       profile     appliances       2020-04-08 12:48:37  xxxx-xxxx-xxxx SUCCESS  No                         essential   FNM00xxxxx..  2020-04-08 14:36:42  xxxx-xxxx-xxxx SUCCESS  No       Post NDU          detailed,.. FNM00xxxxx..  2020-04-09 13:33:57  xxxx-xxxx-xxxx SUCCESS  No                         detailed    FNM00xxxxx..  2020-04-15 15:05:34  xxxx-xxxx-xxxx SUCCESS  Yes      2020-04-15 Dail.. essential.. FNM00xxxxx..  2020-04-16 15:23:07  xxxx-xxxx-xxxx SUCCESS  No       2020-04-16 Dail.. essential   FNM00xxxxx..  2020-04-16 19:07:41  xxxx-xxxx-xxxx SUCCESS  No       Daily data coll.. essential   FNM00xxxxx..

   Here you can see the dates, id, status, if there are dumps, description, profiles used (svc_dc list_profiles), and the appliance the logs correspond to.
   Get the id and run the list command with it to see the full details, including file locations and so on.
   
   Example of the command (output not shown):

   [SVC:service@xxxxxxx-A user]$ svc_dc list xxxx-xxxx-xxxx

   If the bundle ID contains logs for multiple appliances, the full path to the files on each respective appliance is listed in the output.
    
4. Download the logs. There are various ways to download the log file bundles over cli/ssh from the appliances. Here are a few.
   Using tools like WinSCP
   Due to rbash and other security implementations on PowerStore, certain commands and operations are not allowed, for example commands like cd or protocols like SFTP. So you must select SCP as the connection protocol. You only have access to the home directory which is /home/service/user.  However, as described above, links to the logs are created in the home directory. Drag and drop the respective links which should start the download of the file. An error message may be displayed, which does not affect the download once you press ok.
   Using svc_upload or svc_dc download
   This is a service command to facilitate log download or upload. Usage depends on your environment. See the Service Scripts Guide available from the support pages or use these commands for full details: svc_dc download -h or svc_dc upload -h.
   Using windows utility pscp
   Windows provides a tool called pscp (from the command prompt) that can be used to download or upload logs directly from and to the system. The exact location of the file on the PowerStore appliance is required (use the links in the service user home directory).
   
   Syntax:

   pscp -scp service@PowerStore-ip-address:/path/file C:\local_directory

   If using PowerStoreOS V4, add this to the command: pscp -scp -o MACs=hmac-sha2-512 service@xx.xx.xx.xx:...
   
   Example from your Windows command prompt:

   C:\>pscp -scp service@xx.xx.xx.xx:/home/service/user/powerstore_tag_clusterID_serial_2020-04-08_14-36-42_service-data.tgz C:\LOGS service@xx.xx.xx.xx's password: powerstore_tag_clusterID_.. | 320000 kB | 45714.3 kB/s | ETA: 00:00:05 | 54%

Performance Metrics Archives

Using PowerStore Manager

Generating performance metrics is only possible for systems running PowerStoreOS v3.x and higher.

1. Log in to the PowerStore Manager user interface (UI) with your username and password (requires user with administrator user role).
2. Click the Settings menu or the system name.
3. Go to Metrics Archives under the Support section.
4. Click GENERATE METRICS ARCHIVE > GENERATE.
5. Monitor the progress in the Jobs section of PowerStore Manager.
6. When it finishes, from the Metrics Archives section, select the file that and click DOWNLOAD.

Using ssh/cli

For PowerStoreOS version 4.1 and above, use:

For PowerStoreOS version 3.x and 4.0.x:

Requires the admin account or an account with an administrator user role. 

More options can be found with:

Either on or off node PSTCLI versions can be used. For full details on how to use PSTCLI, see the PowerStore CLI guide available at the dell support pages.

The generated performance archive is found inside the following directory, where xxxx is a date folder or folders specific to your system.

The file format is PowerStore_ClusterID_StartDate_EndDate_support-metrics.tar.gz.
Example: PowerStore_abcdefg_2020-04-21T12-56-32_2020-04-28T12-56-32_support-metrics.tar.gz
 

Download the file to a local machine directly with off-node PSTCLI, pscp as shown above, or copy the file to /home/service/user in order to download with WinSCP.

Using off-node PSTCLI:
Example using the off-node PSTCLI on a windows machine with the ID from above:

The file is downloaded to the specified path (-d). If no path is specified, the file is downloaded to where your CMD path defaults to (this can be seen in the command prompt itself within CMD, example: C:\>Users\userX).

Using on-node PSTCLI or direct copy to download with WinSCP:
If running PSTCLI from the primary node directly, copy the file to the home directory, and then transfer with WinSCP.
This can be done with the same command that is shown above (without the second -d switch or with -d /home/service/user).

You can also manually copy the file to /home/service/user as shown here:
List the files available by running

to determine the unique sub directory name. And then:

Copy the file to your current directory (/home/service/user). The dot (.) at the end of the command denotes the target directory:

Once the file is copied, download it with WinSCP.

Dump files

Using PowerStore Manager

Dump files are available from the PowerStore Manager like the Support Materials. Select the relevant entry and press DOWNLOAD. The only difference is that they are listed as Support material dump instead of Support material collection.

Using ssh/cli

The easiest way to find and download dump files over ssh/cli is by logging in to the relevant appliance as service user, and listing these files in the home directory. Like the support materials, links are created here for your convenience.

List or view these directly with WinSCP, or run commands such as ls *dump* to list all available dump files in the home directory.

If the dump files are not listed in the home directory, use the service commands svc_dc list or svc_dc list_dumps. The same logic and principles that are used above for the support materials apply here.

When you get the ID of the dump file, use svc_dc list <id> to get thee exact location. The same download mechanisms can be used. Choose the method that is more convenient.

PowerStore X (Hypervisor logs)

Using PowerStore Manager

The only way to collect hypervisor logs from the PowerStore Manager is by selecting the option under Advanced support materials collection options as explained in step 5 of the above section: Generating a Support Materials log bundle. However, this is not the preferred method as selecting the advanced options greatly increases the size of the log bundle.

Using ssh/CLI

NOTE: vCenter logs are not collected this way. See the next section (Using vSphere).
Once the support material bundle is downloaded, the specific hypervisor logs can be found under:

From vSphere

To collect the hypervisor logs (vm-support) from the vSphere side, see VMware KB 653.
If the vCenter logs are also required for troubleshooting, select the relevant options as explained in the VMware KB.

PowerStore Metro Witness

Log in to the metro witness virtual machine over ssh, and run the built-in data_collect.sh script in order to capture the logs:

Example:

The last line indicates the location of the generated logs.
Download them using something like WinSCP.

Usage:

PowerStore UI/REST logs

Using PowerStore Manager

1. Reproduce the issue in the PowerStore user interface (UI).
2. Hover over the ? help icon in the upper right corner of the UI and press Ctrl + Alt + Shift + Left click. This causes a hidden menu option to appear in the Help menu.
3. Select the Download Network Requests option from that menu and save the text file to a location on your local system.

Using the Internet Browser

1. With the browser open, before launching PowerStore Manager, press F12 to bring up the browser developer window. 
2. Go to the Network tab.
3. Launch the UI and reproduce the issue.
4. In the developer window, right-click any network request in the center panel then select save all as HAR with content.

Switch Logs

This section provides brief guidance on OS10 Dell switches only (ToR). Full details see: How to generate and collect Support bundle in DELL EMC Networking OS10 Switches. For other versions or switch vendors, consult the appropriate documentation.

To start the log collection:

Monitor until they are completed as explained in the KB article above.
Once finished, if you do not have a way to download the logs from the switches, Dell Technical Support can provide an FTP temporary link.

Use these commands to find and download the log bundle:

Where "temp_username@xyz.dell.com" are the details that are provided by Dell Tech Support if you do not have an FTP or SFTP server. The FTP address may differ based on your location.

External storage import logs

Linux host plug-in

Windows host plug-in

EqualLogic

VPLEX and metro node

VPLEX: How to collect logs from a VPLEX