Dell Automated Platform: How to Retrieve a Log Bundle for Troubleshooting

This tool is created to collect all Dell Automation Platform Kubernetes logs for On Prem installations. It also collects native Kubernetes logs and collects pod description in their current states. It collects system log and journald log as well. After collection, it creates a tarball to provide to the engineering team. It is not designed to be used for Software as a Service (SAAS) Dell Automation Platform installations.

This tool traverses  Dell Automation Platform and Knative pods and lists the containers inside. See example below for hzp-product-svc:

root@labmachine01:/nas/tools# kubectl get po -n hzp hzp-product-svc-00001-deployment-5695678c66-79zwm -o jsonpath="{.spec.containers[*].name}"
hzp-ese-svc hzp-product-svc queue-proxy hzp-iam-proxy

For example, the user gets the following file structure inside the tarball:

root@labmachine01:/nas/tools# tar -tf /tmp/native_edge_eo-logs_default_2023-08-30T07-13-39.750/native_edge_eo-logs_default_2023-08-30T07-13-39.750.tar.gz | grep product
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-iam-proxy.current.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-product-svc.current.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-product-svc-00001-deployment-5695678c66-79zwm.hzp-iam-proxy-init.init.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-product-svc-00001-deployment-5695678c66-79zwm.describe
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-ese-svc.previous.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/queue-proxy.current.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/queue-proxy.previous.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-ese-svc.current.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-product-svc.previous.log
./native_edge_eo-logs_default_2023-08-30T07-13-39.750/hzp/hzp-product-svc-00001-deployment-5695678c66-79zwm/hzp-iam-proxy.previous.log

So these files are stored in the following structure:

namespace/pod-name/namespace-container.log

Usage:
First the user must connect and log in into your Dell Automation Platform virtual machine.

Then the user must enable root privileges (sudo -i or su if enabled).

To collect all relevant logs, the user must download the attachment called logs_pkb.zip from this article to the Dell Automation Platform virtual machine.

Extract the contents of the archive:

unzip logs_pkb.zip -d logs-tool

Go to that directory and run the script:

./logs.sh

After execution, the user sees an informational message about available space and a prompt to proceed (enter 'y' to continue):

This script will access your local k3s secrets (secret values are not going to be printed).
This script will collect all logs from all pods in NativeEdge-controlled namespaces and archive it in tarball.
Please press "y" if you agree with this.
You have 69G available on your storage. Consider that logs may require to have more that 1Gb of free space.

Starting from version 1.0-rc logs collection tool detects if NativeEdge or Dell Automation Platform is installed into a non-default namespace. If tool does not find a standard namespace, it prompts the user to enter a valid Orchestrator namespace. See example below:

Available namespaces:
- default
- nativeedge
- kube-node-lease
- kube-public
- kube-system
- longhorn-system
- metallb-system
Warning: Namespace 'hzp' not found.
Please enter the namespace containing the 'nativeedge' containers:

If there is default namespace installation, the tool going to show which namespaces are in the list for collection:

Namespace 'hzp' found. Logs will be collected from following namespaces:
- hzp
- kube-system
- istio-system
- knative-eventing
- knative-serving
- longhorn-system

Events capturing:

Starting from version 0.9-beta of the tool, the user is prompted about EdgeOrchestrator/Dell Automation Platform events capture:

Do you want to collect the EO events (WARNING: this may require significant space on disk)?

This captures events from the EdgeOrchestrator/Dell Automation Platform directly to the logs tarball and improves the troubleshooting. It is advised to get EdgeOrchestrator/Dell Automation Platform events in you have enough space on the machine.

In case the user has additional instructions, the user can check available options of the script before running:

./logs.sh -h

The user can check current version with:

logs.sh -v

This tool is usable on ALL On Prem Dell Automation Platform installations versions, 1.0 and greater.
This tool is usable on ALL NativeEdge Orchestrator versions,  2.0 and greater.