Force 10 - VLT failover testing recommendation and the effect of the failover testing by shutting all the interfaces except the management interface

This document is applicable to all the Dell networking switches that support VLT.

Recommended VLT failover testing

VLT is better known for it's HA(High availability) feature during the interface or switch failure.

In General, management interfaces IPs of VLT peer switches are configured as VLT backup-destination. This backup link maintains heartbeat messages across an out-of-band management network. The backup link only ensures that node failure conditions are correctly detected.

During failover testing, it is common mistake that the administrator shutdown all the interfaces manually, except the management. This active management link allows VLT heartbeat exchange between the VLT peers though VLT peer status is down.

In the above setup, if the both VLTi (Po-1) and VLT Po (Po-2) are shut down on the VLT primary, network administrators may assume that the traffic will be switched to vlt secondary. But this failover will not happen. Instead, in this scenario the VLT Po (Po-2) on VLT secondary will also get shut down automatically and cause a complete network outage.

 VLT-Secondary#do show vlt br

 VLT Domain Brief

------------------

 Domain ID:                  1

 Role:                           Secondary

 Role Priority:                2

 ICL Link Status:            Down

 HeartBeat Status:         Up

 VLT Peer Status:           Link Down

 Local Unit Id:               1

 VLT-Secondary#do show vlt detail

Local LAG Id  Peer LAG Id  Local Status  Peer Status  Active VLANs

------------  -----------  ------------  -----------  -------------

2             2            DOWN          DOWN          10, 20

Only if the heartbeat exchanges are stopped, the VLT-secondary will detect the peer failure and bring up its VLT Port channel (Po2). This is a behavior by design, to avoid split-brain conditions when both the VLT peer switches are active with "Link Down" peer status.

VLT-Secondary(conf)#int man 0/0

VLT-Secondary (conf-if-ma-0/0)#shut

VLT-Secondary (conf-if-ma-0/0)#1d4h57m: %STKUNIT0-M:CP %IFMGR-5-ASTATE_DN: Changed interface Admin state to down: Ma 0/0

1d4h57m: %STKUNIT0-M:CP %IFMGR-5-OSTATE_DN: Changed interface state to down: Ma 0/0

VLT-Secondary (conf-if-ma-0/0)#1d4h57m: %STKUNIT0-M:CP %VLTMGR-6-VLT_HBEAT_DOWN: Heart beat link is down (down).

1d4h57m: %STKUNIT0-M:CP %VLTMGR-6-VLT_ELECTION_ROLE: Chassis is transitioning to primary role.

1d4h57m: %STKUNIT0-M:CP %VLTMGR-6-VLT_PEER_STATUS: Peer chassis is down.

1d4h57m: %STKUNIT0-M:CP %IFMGR-5-OSTATE_UP: Changed interface state to up: Po 2

Conclution:

To perform VLT failover testing, follow the any one of the below steps.

1. Shutdown VLT Port-channel on one of the peer.

2. Reboot one of the VLT peer.

3. Shutdown VLTi, VLT Port-channel and Management interface(or any interface used to reach VLT backup destination) on  the VLT peer.