ECS: OBS: How to generate SupportAssist/TSR logs through racadm or iDRAC UI
Summary: ECS: How to use racadm command to generate SupportAssist logs. How to gather Technical Support Report (TSR) logs from iDRAC UI.
Instructions
Overview
ECS Hardware Models - Gen 3:
- EX300
- EX500
- EXF900
- EX3000 (EX3000S, EX3000D)
ObjectScale Hardware Models - Gen 4:
- X560
- XF960
- EX5000 (EX5000S, EX5000D)
svc_collect profiles feature:
IMPORTANT: As of xDoctor 4.8-86 or later the svc_collect profile log collection feature is available to run on any ECS. If you do not have xDoctor 4.8-86, or later installed then you have to use the manual log collection process. The log generates TSR logs for all models, PERC/RAID Controller logs (EX3000), PERC/RAID Controller Term logs (EX3000) and PERC/RAID Controller Events (EX3000) for PowerEdge EX-Series ECS. More collections must be done manually from the PowerEdge server as requested by the PowerEdge team.
- Run the log gathering command from svc_collect to run for the node you are troubleshooting.
Command:
# svc_collect start -p ic_poweredge -n r<rack number>n<node number>
Example gathering from rack 1 node 1:
Note: If you run the command without identifying the node, it runs on the whole VDC.
admin@node2:~> svc_collect start -p ic_poweredge -n r1n1 svc_collect v1.3.2 (svc_tools v2.6.0-a3) Started 2022-07-29 21:13:02 ---------------------------------------------------------------------------------------------------------------- Starting data collection for: VDC1 Using collection profile: ic_poweredge Collect only from nodes: 'r1n1' Collect configuration files: True Collect log files: True Collect command outputs: True Log file Start time: 2022-07-29 20:53:02 Log file Stop time: 2022-07-29 21:13:02 Maximum root disk util %: 80 Maximum collect size: 10240MB Collect file name: /tmp/svc_collect-ic_poweredge-VDC1-20220729_211302.zip Creating manifest file /tmp/svc_collect-VDC1-20220729_211302/MANIFEST.INF Collecting node command output from nodes [Gathering Service Tag] [Gathering raid log EX3000 only] [Gathering term log] [Gathering events log] [Generating TSR log] [Gathering TSR log - waiting up to 10 minutes to complete] Exporting TSR: SUCCESS Collecting Fabric and OS logs Getting logs for *.raid.log* on node r1n1 Getting logs for *.term.log* on node r1n1 Getting logs for *.events.log* on node r1n1 Getting logs for *_tsr_report.zip* on node r1n1 Collect saved as file /tmp/svc_collect-ic_poweredge-VDC1-20220729_211302.zip Runtime: 40.0968358517
- Collect with a Secure Copy Protocol (SCP) client (WinSCP) the .zip generated from the collection ran, then extract the files with the Dell Service Tag (DST), collections in the file for review by the PowerEdge team.
Manual Steps (before xDoctor 4.8-86)
Command racadm (Remote Access Controller Admin) is a Dell command-line tool loaded on all ECS EX-Series nodes which enables local (ECS Host to iDRAC pass through) and remote (iDRAC physical interface) access to Integrated Dell Remote Access Controller (iDRAC) resources.
SupportAssist logs can be generated if connected locally (on-site) or remotely (SCG or WebEx) using racadm. The iDRAC UI is only available locally because on EX-Series the iDRAC physical interface defaults to using the 192.168.219.10<node#> local IP addresses.
How to generate SupportAssist (SA) logs on an EX300, EX500, X560, EXF900, XF960, EX5000:
Using racadm Locally to connect to iDRAC (ECS Host operational, ssh to target node). Note: EX3000 uses a difference iDRAC version to the above models so refer to step further in this KB:
1. Connect to the target node using NAN IP.
admin:~> ssh 169.254.1.6
2. Run the racadm command to create the TSR collection of logs.
admin:~> sudo -i racadm supportassist collect Job ID = JID_xxxxxx RAC1154: The requested operation is initiated. Run the RACADM jobqueue sub-command, using the job id to check the status of the requested operation.
If you run into a "EULA not accepted" message:
admin@node1:~> sudo -i racadm supportassist collect
SRV085: Unable to start the operation because the SupportAssist End User License Agreement (EULA) is not accepted.
Accept the SupportAssist End User License Agreement (EULA) by running racadm supportassist accepteula command.
A workaround is to use the steps of the section: "Using racadm Remotely to Connect to iDRAC (ECS Host down)", see below.
admin@node2:~> sudo -i racadm -r 192.168.219.101 -u root -p passwd techsupreport collect Security Alert: Certificate is invalid - self signed certificate Continuing execution. Use -S option for racadm to stop execution on certificate-related errors. Job ID = JID_8xxxx RAC1154: The requested operation is initiated. Run the RACADM jobqueue sub-command, using the job id to check the status of the requested operation.
Then go to the "Using racadm Remotely to Connect to iDRAC (ECS Host down)" section as required to extract the logs.
If you run into a "One Instance of Local RACADM is already executing" message:
admin@node1:~> sudo -i racadm supportassist collect One Instance of Local RACADM is already executing. Exiting the current session.
Then see the link for more information: https://www.dell.com/community/Systems-Management-General/RACADM-is-already-executing-message/td-p/4732148
You would need to clear the running racadm process and then rerun your racadm.
Note, check that nobody else requires to run racadm at that moment as you would clear the racadm process then.
The racadm process might be stuck and must be cleared with the following command on the node:
admin@node1:~> sudo rm -rf /var/run/local_racadm.pid
Then rerun your racadm commands.
Stuck racadm processes are more likely on older iDRAC firmware.
If encountered, check the iDRAC firmware level and advise a firmware update if appropriate to do so.
3. The job created by the collection request must complete successfully prior to copying it to the ECS node. Run the following command using the Job ID provided in the preview step to display the job status.
admin:~> sudo -i racadm jobqueue view -i JID_8xxxxx ---------------------------- JOB ------------------------- [Job ID=JID_8xxxxx] Job Name=SupportAssist Collection Status=Completed Start Time=[Not Applicable] Expiration Time=[Not Applicable] Message=[SRV088: The SupportAssist Collection Operation is completed successfully.] Percent Complete=[100] ----------------------------------------------------------
4. Obtain the Dell Service Tag (DST) of the node to be used in the next step.
admin:~> sudo -i racadm getsvctag Cxxxxx
5. Copy the TSR report from iDRAC to directory location with the name <DST#>_tsr_report.zip on the ECS node. The export of the logs is in .zip format by default. The export may take a few minutes.
admin:~> sudo -i racadm supportassist exportlastcollection -f /home/admin/Cxxxxx_tsr_report.zip Support Assist exported successfully. admin:~> ls -alt /home/admin/Cxxxxx_tsr_report.zip -rwxr-xr-x 1 root root 943054 Nov 5 21:07 /home/admin/xxxxx_tsr_report.zip
6. As needed, using SCP or other transfer tools copy the /home/admin/Bxxxxx_tsr_report.zip file to an external location.
Using racadm Remotely to Connect to iDRAC (ECS Host down):
1. Connect to any operational node in the rack where the target nodes reside.
admin:~> ssh 169.254.1.1
Note: By default iDRAC physical network can only be accessed from another node connected to the same backend network (rack).
2. Run racadm command remotely on the target 192.168.219.10<node#> IP using the iDRAC root login/password to create the TSR collection of logs.
admin:~> sudo -i racadm -r 192.168.219.106 -u root -p passwd techsupreport collect Job ID = JID_4xxxxxx RAC1154: The requested operation is initiated. Run the RACADM jobqueue sub-command, using the job id to check the status of the requested operation.
3. The job created by the collection request must complete successfully prior to copying it to the ECS node.
Run the following command using the Job ID provided in the preview step to display the job status.
admin:~> sudo -i racadm -r 192.168.219.106 -u root -p passwd jobqueue view -i JID_4xxxxxxx ---------------------------- JOB ------------------------- [Job ID=JID_4xxxxxxx] Job Name=TSR_Collect Status=Completed Start Time=[Not Applicable] Expiration Time=[Not Applicable] Message=[SYS018: Job completed successfully.] Percent Complete=[100] ----------------------------------------------------------
4. Obtain the Dell Service Tag (DST) of the node to be used in the next step.
admin:~> sudo -i racadm -r 192.168.219.106 -u root -p passwd getsvctag CX1TBM2
5. Copy the TSR report from iDRAC to directory location with the name <DST#>_tsr_report.zip on the ECS node. The export of the logs is in .zip format b default. The export may take a few minutes.
admin:~> sudo -i racadm -r 192.168.219.106 -u root -p passwd techsupreport export -f /home/admin/Cxxxxx_tsr_report.zip Progress = 100 percent Tech Support Report exported successfully. admin:~> ls -alt /home/admin/Cxxxxx_tsr_report.zip -rwxr-xr-x 1 root root 447811 Oct 31 18:42 /home/admin/Cxxxxxx_tsr_report.zip
6. As needed, using SCP or other transfer tools copy the /home/admin/Cxxxxxx_tsr_report.zip file to an external location.
7. To view EX3000 TSR information: Once transferred to a windows laptop, open xxx_tsr_report.zip file in Windows, displays files one of which will be TSR<Numbers>_<DST#>.pl.zip. Open this file, open tsr directory and double-click "viewer.html.
How to capture Tech Support Report (TSR) logs on an EX3000:
Using racadm Locally to connect to iDRAC (ECS Host operational, ssh to target node). Note: EX3000 uses a difference iDRAC version to the above models hence the steps below:
1. Connect to the target node using NAN IP.
admin:~> ssh 169.254.1.6
2. Run the racadm command to create the TSR collection of logs.
admin:~> sudo -i racadm techsupreport collect Job ID = JID_xxxxxx RAC1154: The requested operation is initiated. Run the RACADM jobqueue sub-command, using the job id to check the status of the requested operation.
3. The job created by the collection request must complete successfully prior to copying it to the ECS node.
Run the following command using the Job ID provided in the preview step to display the job status.
admin:~> sudo -i racadm jobqueue view -i JID_xxxxx ---------------------------- JOB ------------------------- [Job ID=JID_xxxxxx] Job Name=TSR_Collect Status=Completed Start Time=[Not Applicable] Expiration Time=[Not Applicable] Message=[SYS018: Job completed successfully.] Percent Complete=[100] ----------------------------------------------------------
4. Obtain the Dell Service Tag (DST) of the node to be used in the next step.
admin:~> sudo -i racadm getsvctag xxxxxx
5. Copy the TSR report from iDRAC to directory location with the name <DST#>_tsr_report.zip on the ECS node. The export of the logs is in .zip format b default. The export may take a few minutes.
admin:~> sudo -i racadm techsupreport export -f /home/admin/xxxxxx_tsr_report.zip Tech Support Report exported successfully. admin:~> ls -alt /home/admin/xxxxxxx_tsr_report.zip -rwxr-xr-x 1 root root 447811 Oct 31 18:42 /home/admin/xxxxxx_tsr_report.zip
6. As needed, using SCP or other transfer tools copy the /home/admin/xxxxxxx_tsr_report.zip file to an external location.
7. To view EX3000 TSR information: Once transferred to a windows laptop, open xxx_tsr_report.zip file in Windows, which displays files one of which will be TSR<Numbers>_<DST#>.pl.zip. Open this file, open tsr directory and double-click "viewer.html.
Note: If this is for a disk-related issue on an EX3000 a request for megacli logs will be requested and follow this KB to collect this information: KB ECS: EX3000: How to collect RAID controller logs
Using racadm Remotely to connect to iDRAC (ECS Host down):
1. Connect to any operational node in the rack where the target nodes reside.
admin:~> ssh 169.254.1.1
Note: By default iDRAC physical network can only be accessed from another node connected to the same backend network (rack).
2. Run racadm command remotely on the target 192.168.219.10<node#> IP using the iDRAC root login/password to create the TSR collection of logs.
admin:~> sudo -i racadm -r 192.168.219.106 -u root -p passwd techsupreport collect Job ID = JID_xxxxx RAC1154: The requested operation is initiated. Run the RACADM jobqueue sub-command, using the job id to check the status of the requested operation.
3. The job created by the collection request must complete successfully prior to copying it to the ECS node.
Run the following command using the Job ID provided in the preview step to display the job status.
admin:~>sudo -i racadm -r 192.168.219.106 -u root -p passwd jobqueue view -i JID_xxxxx ---------------------------- JOB ------------------------- [Job ID=JID_xxxxxx] Job Name=TSR_Collect Status=Completed Start Time=[Not Applicable] Expiration Time=[Not Applicable] Message=[SYS018: Job completed successfully.] Percent Complete=[100] ----------------------------------------------------------
4. Obtain the Dell Service Tag (DST) of the node to be used in the next step.
admin:~> sudo -i racadm -r 192.168.219.106 -u root -p passwd getsvctag xxxxxx
5. Copy the TSR report from iDRAC to directory location with the name <DST#>_tsr_report.zip on the ECS node. The export of the logs is in .zip format b default. The export may take a few minutes.
admin:~> sudo -i racadm -r 192.168.219.106 -u root techsupreport export -f /home/admin/xxxxxx_tsr_report.zip Progress = 100 percent Tech Support Report exported successfully. admin:~> ls -alt /home/admin/xxxxxxx_tsr_report.zip -rwxr-xr-x 1 root root 447811 Oct 31 18:42 /home/admin/xxxxxxx_tsr_report.zip
6. As needed, using SCP or other transfer tools copy the /home/admin/xxxxxx_tsr_report.zip file to an external location.
7. To view EX3000 TSR information: Once transferred to a windows laptop, open xxx_tsr_report.zip file in Windows, which displays files one of which will be TSR<Numbers>_<DST#>.pl.zip. Open this file, open tsr directory and double-click "viewer.html.