PowerFlex: Todos os SDCs desconectados da vmnic do SDS foram redefinidos para half-duplex e negociados automaticamente.
Summary: Nesse caso, o VxRack-Flex estava tendo seu RCM atualizado.
Symptoms
Durante o upgrade da RCM de um VxRack Flex, um servidor perdeu a comunicação entre seu SDS e todos os SDCs.
Cause
Por algum motivo, o duplex e a velocidade no vmnic0 foram redefinidos para half-duplex e negociados automaticamente.
Quando o sistema foi ligado, o Scaleio tentou conduzir a E/S por essa NIC, e ele continuou falhando porque o Scaleio estava tentando usar full duplex e velocidade 10000.
Isso fez com que o switch Cisco desligasse a porta vmnic0 estava ligada, pois o switch achava que a porta estava oscilando devido a um erro de rede.
Aqui estão alguns registros do vmkernel.log demonstrando o abalo causado por esse cenário:
2019-05-01T15:46:03.721Z cpu26:86039)netschedHClk: NetSchedHClkWatchdogSysWorld:4552: vmnic0: link up event received, device running at 10000 Mbps so setting queue depth to 86460 bytes with expected 1310 bytes/us 2019-05-01T15:46:04.648Z cpu42:85995)netschedHClk: NetSchedHClkWatchdogSysWorld:4364: vmnic0: hclk scheduler instance clean up 2019-05-01T15:46:04.649Z cpu45:85995)netschedHClk: NetSchedHClkDoFlushQueue:3874: vmnic0: dropping 42 packets from queue netsched.pools.persist.default 2019-05-01T15:46:04.649Z cpu45:85995)netschedHClk: NetSchedHClkDoFlushQueue:3874: vmnic0: dropping 501 packets from queue netsched.pools.vm.67108869 2019-05-01T15:46:04.650Z cpu45:85995)netschedHClk: NetSchedHClkDoFlushQueue:3874: vmnic0: dropping 122 packets from queue netsched.pools.persist.mgmt 2019-05-01T15:46:04.653Z cpu45:85995)netschedHClk: NetSchedHClkWatchdogSysWorld:4475: vmnic0: watchdog world (worldID = 85995) exits 2019-05-01T15:46:07.015Z cpu42:66250)ixgben: ixgben_CheckTxHang:1762: vmnic0: false hang detected on TX queue 0 2019-05-01T15:46:12.017Z cpu42:66250)ixgben: ixgben_CheckTxHang:1762: vmnic0: false hang detected on TX queue 0 2019-05-01T15:46:12.649Z cpu38:65725)ixgben: indrv_UplinkReset:1520: indrv_UplinkReset : vmnic0 device reset started 2019-05-01T15:46:12.649Z cpu38:65725)ixgben: indrv_UplinkQuiesceIo:1483: Stopping I/O on vmnic0 2019-05-01T15:46:12.740Z cpu52:66252)ixgben: ixgben_CheckLink:2514: Link is down for device vmnic0 (0x4307f4af4540) 2019-05-01T15:46:12.740Z cpu52:66252)netschedHClk: NetSchedHClkNotify:2908: vmnic0: link down notification 2019-05-01T15:46:12.740Z cpu52:66252)netschedHClk: NetSchedHClkDoFlushQueue:3874: vmnic0: dropping 211 packets from queue netsched.pools.vm.67108869 2019-05-01T15:46:12.740Z cpu38:65725)ixgben: indrv_DeviceReset:2382: Device Resetting vmnic0 2019-05-01T15:46:12.740Z cpu38:65725)ixgben: indrv_Stop:1950: stopping vmnic0 2019-05-01T15:46:13.013Z cpu38:65725)ixgben: indrv_UplinkStartIo:1460: Starting I/O on vmnic0 2019-05-01T15:46:13.130Z cpu38:65725)ixgben: indrv_UplinkReset:1540: indrv_UplinkReset : vmnic0 device reset completed 2019-05-01T15:46:13.177Z cpu52:66252)ixgben: ixgben_CheckLink:2514: Link is up for device vmnic0 (0x4307f4af4540) 2019-05-01T15:46:13.177Z cpu52:66252)netschedHClk: NetSchedHClkNotify:2900: vmnic0: link up notification 2019-05-01T15:46:13.341Z cpu52:66252)ixgben: ixgben_CheckLink:2514: Link is down for device vmnic0 (0x4307f4af4540) 2019-05-01T15:46:13.341Z cpu52:66252)netschedHClk: NetSchedHClkNotify:2908: vmnic0: link down notification 2019-05-01T15:46:13.348Z cpu52:66252)ixgben: ixgben_CheckLink:2514: Link is up for device vmnic0 (0x4307f4af4540) 2019-05-01T15:46:13.348Z cpu52:66252)netschedHClk: NetSchedHClkNotify:2900: vmnic0: link up notification 2019-05-01T15:46:13.573Z cpu52:66252)ixgben: ixgben_CheckLink:2514: Link is down for device vmnic0 (0x4307f4af4540) 2019-05-01T15:46:13.573Z cpu52:66252)netschedHClk: NetSchedHClkNotify:2908: vmnic0: link down notification
Resolution
1) Defina as configurações corretas na porta vmnic Nesse caso, era full-duplex e velocidade 10000.
2) Recupere a porta no switch Cisco:
Putty para o switch Cisco que possui a porta que deve ser devolvida.
Exiba todas as portas no switch:
#show interfaces status
No nosso caso, esse foi o status que vimos que era ruim:
Eth1/2/3 316 eth access down linkFlapErrDisabled auto(D) --
Agora vamos pular a porta:
# enable # configure terminal (config)# interface Ethernet 1/2/3 (config-subif)# shutdown (config-subif)# no shutdown (config-subif)# end # show interfaces Ethernet 1/2/3 status
A porta mostrará que está ativada e o problema com o SCALEIO será corrigido assim que começar a conduzir E/S novamente.