Metro node: collect-diagnostics command run on multiple nodes of a metro node cluster simultaneously causes missing data in the gathered logs

Summary: This article talks to the issue when a user runs the command 'collect-diagnostics' on two different nodes of a Metro Node cluster at the same time, the gathered log data will be missing some logs and data. Workaround provided. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

IMPACTED DELL METRO NODE SOFTWARE:  
Metro Node OS 7.0
Metro Node OS 7.1 Patch 1
Metro Node  8.0
 

IMPACTED DELL METRO NODE HARDWARE: 
Metro Node-mn114
Metro Node-mn215


NOTE: The term "node" may be used in this article at times and refers to the "directors" in a Metro Node.
 

Issue:

When the command 'collect-diagnostics' is executed when another instance of 'collect-diagnostics' was already running on another node, of the Metro Node, this causes data in the gathered CD* logs to be missing in one of the outputs of the collect-diagnostics.
 

*NOTE: CD/CDs are the shortened term used for collect-diagnostics when seen in this article.
 

Symptom:

Refer to the examples below that show the differences in the output between the two different collect-diagnostics logs collected after running the CDs through VS5LAT tool on the log server.

The below example shows incomplete log data in the parsed collect-diagnostics logs: 
[vplexuser@vplexlogserver 4Kxxx94-c2-diag-2022-12-06-11.49.44]$ ll
total 284
drwxr-xr-x 2 vplexuser logserverusers   4096 Dec  7 03:40 analysisnotes
-rw-r--r-- 1 vplexuser logserverusers 276480 Dec  7 03:40 AnalysisNotes-4Kxxx94-c2-diag-2022-12-06-11.49.44.tar.tar
-rw-r--r-- 1 vplexuser logserverusers   5752 Dec  7 03:40 AnalysisNotes.html
drwxr-xr-x 2 vplexuser logserverusers    189 Dec  7 03:40 cdlogs
drwxr-xr-x 2 vplexuser logserverusers     10 Dec  7 03:40 cws
drwxr-xr-x 2 vplexuser logserverusers     65 Dec  7 03:40 director-1-1-A
drwxr-xr-x 2 vplexuser logserverusers     65 Dec  7 03:40 director-1-1-B
drwxr-xr-x 2 vplexuser logserverusers     65 Dec  7 03:40 director-2-1-A
drwxr-xr-x 2 vplexuser logserverusers     65 Dec  7 03:40 director-2-1-B
drwxr-xr-x 3 vplexuser logserverusers     83 Dec  7 03:40 mgmt_server
drwxr-xr-x 2 vplexuser logserverusers    270 Dec  7 03:40 misc
drwxr-xr-x 4 vplexuser logserverusers     61 Dec  7 03:40 performance
drwxr-xr-x 2 vplexuser logserverusers     58 Dec  7 03:40 syrconfig

The below example shows the expected output to be seen from the parsed collect-diagnostics logs: 
[vplexuser@vplexlogserver 4Kxxx96-c1-diag-2022-12-12-08.40.52]$ ll
total 420
drwxr-xr-x  2 vplexuser logserverusers   4096 Dec 13 01:58 analysisnotes
-rw-r--r--  1 vplexuser logserverusers 419840 Dec 13 01:58 AnalysisNotes-4Kxxx96-c1-diag-2022-12-12-08.40.52.tar.tar
-rw-r--r--  1 vplexuser logserverusers   3962 Dec 13 01:58 AnalysisNotes.html
drwxr-xr-x  2 vplexuser logserverusers    189 Dec 13 01:58 cdlogs
drwxrws--- 11 vplexuser logserverusers    172 Dec 12 03:40 collect_director-1-1-a_2022-12-12-08.42.44
drwxrws--- 11 vplexuser logserverusers    172 Dec 12 03:47 collect_director-1-1-b_2022-12-12-08.49.07
drwxrws--- 11 vplexuser logserverusers    172 Dec 12 03:53 collect_director-2-1-a_2022-12-12-08.55.17
drwxrws--- 11 vplexuser logserverusers    172 Dec 12 03:59 collect_director-2-1-b_2022-12-12-09.01.27
drwxr-xr-x  3 vplexuser logserverusers    144 Dec 13 01:58 director-1-1-A
drwxr-xr-x  3 vplexuser logserverusers    144 Dec 13 01:58 director-1-1-B
drwxr-xr-x  3 vplexuser logserverusers    144 Dec 13 01:58 director-2-1-A
drwxr-xr-x  3 vplexuser logserverusers    144 Dec 13 01:58 director-2-1-B
drwxr-xr-x  3 vplexuser logserverusers    166 Dec 13 01:58 firmwarelogs
drwxr-xr-x  3 vplexuser logserverusers     31 Dec 13 01:55 mgmt_server
drwxr-xr-x  2 vplexuser logserverusers    270 Dec 13 01:58 misc
drwxr-xr-x  4 vplexuser logserverusers     61 Dec 13 01:58 performance
drwxr-xr-x  6 vplexuser logserverusers    114 Dec 13 01:58 scif
drwxr-xr-x  2 vplexuser logserverusers     58 Dec 13 01:58 syrconfig


 

Cause

The simultaneous running of the 'collect-diagnostics' command, run on different nodes, causes some data/logs to be missed in one of the CD logs.

Resolution

Permanent Resolution:

Engineering is investigating this problem, once a fix is available this article will be updated.
 

Workaround:

To resolve this issue, execute the 'collect-diagnostics' command again but this time from only one director, on only one cluster if a Metro configuration, and wait for its full completion before gathering the CDs from another director or from the peer cluster, if needed. 

Note: Executing the 'collect-diagnostics' command on one director of a metro node cluster should have all the data from all directors from all nodes of a cluster.

Refer to KB Article 000197436 for more details on 'How to collect logs from the metro node '.

Affected Products

metro node mn-114, metro node mn-215
Article Properties
Article Number: 000223610
Article Type: Solution
Last Modified: 02 Apr 2024
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.