Connectrix Cisco MDS: Show tech-support details command stops responding and does not print any output

Summary: 'Show system internal eobcsw' commands produce an error or stop responding forever without any error or output.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

These commands are in common commands such as: 
  • Show tech-support 
  • Show tech-support details 
  • Show tech-support module
    (when a Supervisor 4 is included)

Affected commands stop responding after printing any initial banner. For example:
  
Switch# Show tech-support details
Show tech details will take 4-8 minutes to complete. Please Wait...
(no further output)

Cause

This issue applies only to MDS 9700 Supervisor 4 modules. 
  • The time interval after the boot of the supervisor when the issue can happen is variable. Frequent use of the Show system internal eobcsw commands decrease the time before impact. If these commands are not frequently used, the issue can happen about 60 days after the supervisor boots.
  • A supervisor uptime may be seen with the 
     slot sup_slot show system uptime | include Kernel

    command, where sup_slot is the slot number of a supervisor. For example:        
 Kernel uptime:              61 days, 23 hours, 40 minutes, 42 seconds
 
  • This issue by a memory leak in the mvdxn process supporting the show system internal eobcsw commands. The memory leak occurs at a constant rate which cannot be reduced. Memory utilization of this process can be monitored and the indicator that the issue occurred is when the MemUsed parameter becomes approximately equal to MemLimit.

Example of a system that did not yet encounter the issue:
  
m9710-1# show processes memory | i "PID|mvdxn"

PID    MemAlloc  MemLimit    MemUsed     StackBase/Ptr      Process
7712  167862272  738421939   525918208   fff50d40/fff4fc60  mvdxn


Example of a system that did encounter the issue.
  
m9718-1# show processes memory | i "PID|mvdxn"

PID    MemAlloc  MemLimit    MemUsed     StackBase/Ptr      Process
7282  381943808  738448972   738447360   ffa08c60/ffa07b80  mvdxn


Resolution

Work Around:
  • As mentioned, there is no workaround to the mvdxn process memory leak problem itself. To prevent this issue of the Show tech-support detail stopping responding, it is necessary to reload affected supervisors before the available memory is exhausted. This can be done non-disruptively by:
  1. Identifying the standby supervisor and non-disruptively reloading it with the reload module command
  2. Waiting for the standby supervisor to reach 'ha-standby' state
  3. Performing a non-disruptive system switchover command.
  • If a system switchover cannot be immediately started, but a Show tech-support detail output must be collected, you can use the following EEM configuration to disable the eobcsw commands:
event manager applet BLOCK_EOBCSW
event cli match "show hardware internal eobcsw*"
action 0.1 syslog priority warnings msg"SHOW_HW_INTERNAL_EOBCSW_BLOCKED_CSCwe46073"

Further Problem Description:
  • The impact of this issue is limited to the execution of Show system internal eobcsw commands. It does not impact data traffic and does not prevent operations such as configuration changes, non-disruptive upgrades, or supervisor switchovers.
  • Once the process mvdxn consumes all the memory allocated to it, the behavior of these commands is release-dependent:
  • NX-OS 8.4(2e) - the commands generate an error and complete. Show tech-support type commands complete successfully without the output of the affected commands.
  • NX-OS 9.3(1) and 9.3(2) - the commands stop responding and a command may be aborted with 'control-c' or, if it is an SSH session, by disconnecting.
 
Affected Release:
  • NX-OS v8.4(2e)
  • NX-OS v9.3(1)
  • NX-OS v9.3(2)

Fixed Release:
  • NX-OS v8.4(2f)
  • NX-OS v9.3(2a)
  • NX-OS v9.4(1)


 

Affected Products

Connectrix MDS-9706-V2, Connectrix MDS-9710, Connectrix MDS-9710-V2, Connectrix MDS-9718, Connectrix MDS-9718-V3

Products

Connectrix MDS-Series Firmware, Connectrix MDS-Series Hardware
Article Properties
Article Number: 000228003
Article Type: Solution
Last Modified: 26 Aug 2024
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.