Dell Unity: How to Find and Download DUMP Files From a Unity System (User Correctable)

From Unisphere 

Only available for Unity OE 4.2.0.9392909 and later).
Log in to Unisphere and go to Service > Downloads > Core Dump.
This menu facilitates the retrieval of Core Dump files from either SP regardless of which SP is primary at the time.
Once downloaded from the array, upload the files to the Service Request. If the file is too large, request an FTP location from your support representative. 
 

CLI or SSH

1. Connect to the Unity over SSH and login as service user.
2. Take note of which Storage Processor (SP) you are on, the SP you are automatically connected to is the primary SP. If you are not sure, run pgrep ECOM. The primary SP should return a process ID number, unlike the secondary SP which returns nothing. See Additional Information for an example of this.
3. If your dump file is on the primary SP, then issue these commands:
   • svc_dc -lcd to get a list of the available core dumps on the local SP. Each entry corresponds to a dump file. Find the dump that you need (date, and time is in the dump name).
   • svc_dc -dc /home/service/user <dump file name> to compress and copy the file (There is a space between /home/service/user and the dump name).

4. If your dump file is on the secondary SP, login to the secondary SP (ssh peer) and from there:
   • svc_dc -lcd to list available core dumps on the local SP.
   • svc_dc -dc peer /home/service/user <dump file name> (There is a space between /home/service/user and the dump name.) This compresses the dump folder and moves it to /home/service/user on the peer SP. In this case, we are copying the file to the primary SP, from where we can get it with WinSCP.
   • In the command syntax, we are specifying the target with "peer" which moves the compressed file to the peer SP. This command can also be used to move the resulting file to remote systems over SSH. For more information, see svc_dc -h.
5. Open an SFTP client (such as WinSCP) and login to the management IP address as service. (This connects you to the primary SP only.)
6. Go to the /home/service/user directory (or /cores/service/user which is the same place), download the dump file that you created in steps 3 or 4, and upload to the Service Request. If the file is too large, request an FTP location from your support representative. 
7. Once downloaded, delete the created DUMP files from /home/service/user and the original files from /EMC/backend/service/data_collection/cores/ (requires root, Dell Tech support only).

If any assistance is required, contact Dell Technical Support or your Authorized Service Representative, and quote this article ID.
 

Compressing and copying Core Dump files manually

If you are not familiar with the Linux "tar" command, do not use this method as it may corrupt the dump files, or cause additional issues (like reboot an SP) if files are created in the wrong places. This method should be used only as a last resort if everything else failed and should be used only by advanced Linux users.

If the above method was understood and everything else failed, the DUMP file directory can also be compressed manually. Unity dumps are in /EMC/backend/service/data_collection/cores/ on each SP.

Compress the dump directory to a file, tar -zcvf /home/service/user <destination_filename.tar.gz> <dump_directory_name>.
If doing this on Unity OE older than 4.5, use /home/service, as the next level directory "user" was only created since Unity OE 4.5.

The resulting .tar.gz file is in the /home/service/user directory (or /cores/service/user which is the same place).
For Unity OE earlier than 4.5, the file should be created in /cores/service as described above.

Do not use other target directories! You can cause serious system issues.

If you need logs using this method from the secondary SP, see Dell KB article 22501: Dell Unity: How to Download Files from the Secondary SP Using the Primary SP (User Correctable) on how to get files from the secondary SP with SCP.

Delete any dumps from the following locations on both SPs once the files are downloaded from the array.
/cores/service/user or /home/service/user (/home/service or /cores/service for Unity OE older than 4.5)
/EMC/backend/service/data_collection/cores (requires root - Dell Tech support only).

This is to avoid issues in the future (such as corrupt Core Dump files or the issues that are described in Dell KB article 52543: Dell Unity: Upgrade to Unity OE 4.2 fails with error: check_cores_existed_files_2 (User Correctable) 

Files in /EMC/backend/service/data_collection/cores can only be deleted as root user, so contact Dell Technical Support or your Authorized Service Representative, and quote this article ID.

IMPORTANT: Compare the md5sum of the files (downloaded vs original file on SP) before permanently deleting the files.

Check the original dump file md5sum on /EMC/backend/service/data_collection/cores/<dump_folder>, and not a file you may have compressed manually.
To check md5sum, run these:

Linux:       md5sum YourDumpFile.gz
Windows:     certUtil -hashfile YourDumpFile.gz MD5

 

Examples of the commands described above

Using pgrep to confirm which SP is primary:

service@none spa:~# pgrep ECOM 30065

You get a Process ID (PID) for ECOM, this is the primary SP. ECOM (management) only runs on the primary SP. In this example, SPA is primary.

But as expected, once you ssh into the secondary SP, no PID is returned for ECOM as ECOM does not run on the secondary SP:

service@none spa:~#ssh peer service@none spb:~# pgrep ECOM service@none spb:~#

If the SPs are in service mode, none of the SPs return a PID for ECOM. ECOM does not run if the SPs are in service mode.

Listing, compressing, and moving a dump file on the primary SP:
 

service@(none) spa:/# svc_dc -lcd ======================== [DC copier]: Available on backend: core-dump_dump_spa_CKM0016170xxxx_2017-01-17_15_14_17_320_httpd core-dump_dump_spa_CKM0016170xxxx_2017-02-16_07_19_23_337_xGate kdump_spa_CKM0016170xxxx_2017-01-11_14_52_40 safe_dump_spa_CKM0016170xxxx_2017-02-24_11_26_28_31949_safe << we are using this for the example. ======================== [DC copier]: No known cores at SSD in processed ======================== [DC copier]: Not yet pre-processed core dumps at SSD: ======================== service@(none) spa:/# svc_dc -dc /home/service/user safe_dump_spa_CKM0016170xxxx_2017-02-24_11_26_28_31949_safe [DC copier]: There is a DC bundle in /EMC/backend/service/data_collection/cores/safe_dump_spa_CKM0016170xxxx_2017-02-24_11_26_28_31949_safe_dir. [DC copier]: safe_dump_spa_CKM0016170xxxx_2017-02-24_11_26_28_31949_safe_nogdb.tar will contain: safe_dump_spa_CKM0016170xxxx_2017-02-24_11_26_28_31949_safe_dir [DC copier]: safe_dump_spa_CKM0016170xxxx_2017-02-24_11_26_28_31949_safe_nogdb.tar transmitted to /home/service/user successfully service@(none) spa:/#cd /home/service/user service@(none) spa:/home/service/user# ll (or ls) ... -rw-r--r-- 1 root    root    681431040 Mar  8 22:24 safe_dump_spa_CKM0016170xxxx_2017-02-24_11_26_28_31949_safe_nogdb.tar

 

Listing, compressing, and moving a dump file on the secondary SP:

service@(none) spa:~# ssh peer   << to change to the secondary SP service@(none) spb:~# svc_dc -lcd  << listing all available core dumps on SPB (secondary in this case) ======================== [DC copier]: Available on backend: core-dump_dump_spb_CKM0016170xxxx_2016-09-08_09_18_19_11994_mgmtd <<< we use this for the example. core-dump_dump_spb_CKM0016170xxxx_2017-01-02_13_35_47_10385_grep core-dump_dump_spb_CKM0016170xxxx_2017-01-13_11_38_22_17432_TLDlistener.exe core-dump_dump_spb_CKM0016170xxxx_2017-01-25_09_19_38_338_httpd core-dump_dump_spb_CKM0016170xxxx_2017-01-27_03_15_52_2648_xGate ======================== [DC copier]: No known cores at SSD in processed ======================== [DC copier]: Not yet pre-processed core dumps at SSD: ======================== service@(none) spb:~# svc_dc -dc peer /home/service/user core-dump_dump_spb_CKM0016170xxxx_2016-09-08_09_18_19_11994_mgmtd [DC copier]: There is a DC bundle in /EMC/backend/service/data_collection/cores/core-dump_dump_spb_CKM0016170xxxx_2016-09-08_09_18_19_11994_mgmtd_dir.  [DC copier]:   you have to enter credentials within 15 seconds or progress counter will start [DC copier]: core-dump_dump_spb_CKM0016170xxxx_2016-09-08_09_18_19_11994_mgmtd_nogdb.tar will contain: core-dump_dump_spb_CKM0016170xxxx_2016-09-08_09_18_19_11994_mgmtd_dir [DC copier]: core-dump_dump_spb_CKM0016170xxxx_2016-09-08_09_18_19_11994_mgmtd_nogdb.tar transmitted successfully service@(none) spb:~# exit  <<< or "ssh peer" to go back to the primary SP and confirm that the newly created file is there. logout Connection to peer closed. service@(none) spa:~# cd /cores/service/user <<< same place as /home/service/user service@(none) spa:~# ll ... -rw-r--r-- 1 root    root    222310400 Mar  9 14:37 core-dump_dump_spb_CKM001617xxxxx_2016-09-08_09_18_19_11994_mgmtd_nogdb.tar

Our dump file (originally from SPB) is on the primary SP, SPA in this example, and ready for download. Remember to delete any dumps from /cores/service/user or /home/service/user on both SPs once the files are downloaded from the array.

Dell Unity 300, Dell EMC Unity 300F, Dell EMC Unity 350F, Dell EMC Unity 400, Dell EMC Unity 400F, Dell EMC Unity 500, Dell EMC Unity 500F, Dell EMC Unity 550F, Dell EMC Unity 600, Dell EMC Unity 600F, Dell EMC Unity 650F , Dell EMC Unity Family |Dell EMC Unity All Flash, Dell EMC Unity Family, Dell EMC Unity Hybrid ... View More View Less