PowerFlex Communication Attempts Between Segregated Management And Data Networks
Summary: An MDM uses its management (MGMT) interface to transmit an SYN segment over the network to data (DATA) network-related SDS IPs. Network monitoring application or software-based firewall might alert an MDM source sending a SYN segment over an MGMT network to a DATA network target. ...
Symptoms
The management IPs for the MDMs are 10.4.112.138, 10.4.112.140, and 10.4.112.139.
MDM cluster details:
Cluster:
Name: mdm_cluster, ID: 40e0c8ba4c5a090f, Mode: 5_node, State: Normal, Active: 5/5, Replicas: 3/3
Virtual IP Addresses: 10.4.108.144, 10.4.109.16, 10.4.110.144, 10.4.111.16
Master MDM:
Name: MDM1, ID: 0x7436782c15077b00
IP Addresses: 10.4.108.136, 10.4.109.8, 10.4.110.136, 10.4.111.8, Management IP Addresses: 10.4.112.138, Port: 9011, Virtual IP interfaces: em3.108, em4.109, p2p1.110, p2p2.111
Status: Normal, Version: 3.6.500
Actor ID: 0x2c90158170e71800, Voter ID: 0x22678c2f497ecc00
Slave MDMs:
Name: MDM3, ID: 0x2cceb7e13f8bb203
IP Addresses: 10.4.108.138, 10.4.109.10, 10.4.110.138 [D], 10.4.111.10, Management IP Addresses: 10.4.112.140, Port: 9011, Virtual IP interfaces: em3.108, em4.109, p2p1.110, p2p2.111
Status: Normal, Version: 3.6.500
Actor ID: 0x1bf2db164f660203, Voter ID: 0x0f57f19f691a5c03, Replication State: Normal
Name: MDM2, ID: 0x3f64c5d17c27af01
IP Addresses: 10.4.108.137, 10.4.109.9, 10.4.110.137, 10.4.111.9, Management IP Addresses: 10.4.112.139, Port: 9011, Virtual IP interfaces: em3.108, em4.109, p2p1.110, p2p2.111
Status: Normal, Version: 3.6.500
Actor ID: 0x65d2dbf937210301, Voter ID: 0x4b1a93ed3afce101, Replication State: Normal
Tie-Breakers:
Name: MDM5, ID: 0x4da9a5db7d2fab04
IP Addresses: 10.4.108.140, 10.4.109.12, 10.4.110.140, 10.4.111.12, Port: 9011
Status: Normal, Version: 3.6.500
Voter ID: 0x6360d7eb2a73cd04
Name: MDM4, ID: 0x77ff71c05cf58d02
IP Addresses: 10.4.108.139, 10.4.109.11, 10.4.110.139, 10.4.111.11, Port: 9011
Status: Normal, Version: 3.6.500
Voter ID: 0x39acc8f2009a9f02
[D] - represents a network being [D]own, or unresponsive,
The NICs (Physical) and tagged/bonded interfaces (Virtual) are configured to have separate DATA and MGMT networks per interface.
In this scenario, both NICs em3 and p2p2 are bonded under the bond0 interface, and bond0 is configured with the MGMT network vlan.
The vlan network distribution is as follows; the DATA networks are divided between the 108-111 VLANs, and the MGMT network is on VLAN 112.
The MGMT network is running on a virtual interface, bond0, which is a virtual layer of the em3 and p2p2 NICs:
Interface Physical/Virtual MAC address IP address VLAN ID BOND details === === === === === === bond0 Virtual e8:eb:d3:64:cf:a6 bond0.112 Virtual e8:eb:d3:64:cf:a6 10.4.112.140 VLAN 112 em3 Physical e8:eb:d3:64:cf:a6 bond0-sl1 em3.108 Virtual e8:eb:d3:64:cf:a6 10.4.108.138 VLAN 108 em4 Physical e8:eb:d3:64:cf:a7 em4.109 Virtual e8:eb:d3:64:cf:a7 10.4.109.10 VLAN 109 p2p1 Physical e8:eb:d3:71:ac:d6 p2p1.110 Virtual e8:eb:d3:71:ac:d6 10.4.110.138 VLAN 110 p2p2 Physical e8:eb:d3:64:cf:a6 bond0-sl2 p2p2.111 Virtual e8:eb:d3:64:cf:a6 10.4.111.10 VLAN 111
When validating in netstat, one of the MDM cluster nodes (mdm_3) is transmitting SYN segments over the MGMT network to a DATA network IP (VLAN 110) on SDSs and MDMs.
It is worth noticing that this communication is only one-sided since we only see SYN-SENT sent but not acknowledged (SYN-RECEIVED/ACK/ESTABLISHED) by IPs on the DATA network of VLAN 110:
SYN-SENT - represents waiting for a matching connection request after having sent a connection request.
logs# find ./* -type f -name "netstat_pan.txt" | while read -r i; do echo "${i}"; echo "==="; grep -i "SYN" "${i}"; echo ""; done
./mdm_1/server/netstat_pan.txt
===
tcp 0 1 10.4.110.136:48896 10.4.110.138:7072 SYN_SENT 2475/mdm-3.6.500.11
tcp 0 1 10.4.110.136:44608 10.4.110.138:9011 SYN_SENT 2475/mdm-3.6.500.11
tcp 0 1 10.4.110.136:48902 10.4.110.138:7072 SYN_SENT 2475/mdm-3.6.500.11
tcp 0 1 10.4.110.136:48897 10.4.110.138:7072 SYN_SENT 2475/mdm-3.6.500.11
tcp 0 1 10.4.110.136:48900 10.4.110.138:7072 SYN_SENT 2475/mdm-3.6.500.11
./mdm_2/server/netstat_pan.txt
===
tcp 0 1 10.4.110.137:43896 10.4.110.138:9011 SYN_SENT 2535/mdm-3.6.500.11
./mdm_3/server/netstat_pan.txt
===
tcp 0 1 10.4.112.140:57266 10.4.110.137:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:50040 10.4.110.136:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:57250 10.4.110.137:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:50044 10.4.110.136:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:51450 10.4.110.139:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:57252 10.4.110.137:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:50054 10.4.110.136:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:51430 10.4.110.139:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:50050 10.4.110.136:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:52756 10.4.110.137:9011 SYN_SENT 2534/mdm-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:38624 10.4.110.139:9011 SYN_SENT 2534/mdm-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:51442 10.4.110.139:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:38624 10.4.110.140:9011 SYN_SENT 2534/mdm-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:60010 10.4.110.136:9011 SYN_SENT 2534/mdm-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:51446 10.4.110.139:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
tcp 0 1 10.4.112.140:57262 10.4.110.137:7072 SYN_SENT 2535/sds-3.6.500.11 <<<<<
See below examples of a successfully established communication vs. a single SYN sent in the scenario described above:
Successful communication established:
Unsuccessful communication attempt (a single SYN sent):
This is a purely cosmetic impact, as there is no full communication established between the networks, this is not a security issue.
The MGMT source IP is sending a "hello" message to several DATA target IPs (sending an SYN, unlike SYN-ACK, or ACK which tries to or does establish a communication path - see Successful communication-established example above).
Cause
The system is designed to try to check on the DATA network's health by any means necessary. As such, the system tries to go through whichever path it has to check it.
In this scenario, one of the DATA NICs, p2p1, went down, and as a result, the DATA interface, p2p1.110, went down as well. The system reacts to the "loss" of the VLAN 110 network by trying the only path that it has to it, with any network it can, in this case, VLAN 112 goes through any NIC in a UP state.
When validating in route -n, we can see that DATA interface p2p1.110. Also, all network traffic seems to be directed to interface bond0.112, which is also the MGMT network interface, as it is the default route for all traffic.
It is worth mentioning that if the default gateway were configured on a different VLAN, it would have been using that interface to reach the unavailable IPs.
As a result, we see SYN segments sent from VLAN 112 IP address (MGMT) to VLAN 110 address IPs (DATA).
Based on the system design, it intends to check for a "life sign" by any means necessary. Still, the network logic and configuration are set not to allow such a communication to occur and this is why we only see a SYN_SENT sent to any of the IPs on the VLAN 110 network, and no return segment sent back.
See a visual figure of the scenario in the screenshots below.
MGMT and DATA networks run separately, each on its relative VLAN:
NIC p2p1 goes down:
The system sends an SYN segment from the MGMT IP address to the DATA IP addresses to detect a "life sign":
Resolution
Impacted Versions
All PowerFlex versions
Fixed In Version
N/A - expected behavior