PowerFlex 3.X: SDCs werden bei hoher Lese-I/O-Auslastung von den SDS getrennt

Bestimmte SDCs mit einer höheren Lese-I/O-Last werden kurz von den SDS getrennt. 

– Ereignisse aus der MDM-events.txt:

2021-07-10 22:31:22.274 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 
2021-07-10 22:31:23.897 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 
2021-07-10 23:14:42.041 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 
2021-07-10 23:14:44.041 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 
2021-07-10 23:21:13.722 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 
2021-07-10 23:21:15.561 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 
2021-07-10 23:31:09.561 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 
2021-07-10 23:31:11.353 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 
2021-07-11 20:57:22.105 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 
2021-07-11 20:57:23.709 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 
2021-07-11 21:02:20.985 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 
2021-07-11 21:02:21.996 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 
2021-07-11 21:04:58.553 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 
2021-07-11 21:05:00.473 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 

 – Ereignisse auf SDC-Seite:

2021-07-11T21:04:58.553170+00:00 sdc1001 kernel: [19011355.478437] PowerFlex netCon_IsKaNeeded:4321 :CON ffff8c3eb591b900 peer 192.168.1.145 didn't receive message for 30 iterations.  Marking as down
2021-07-11T21:04:58.552199+00:00 sdc1001 kernel: [19011355.491770] [23305989309] Disconnected from SDS with ID f34f1b6800000025
2021-07-11T21:04:59.156173+00:00 sdc1001 kernel: [19011355.598778] [23305989416] Connected to SDS with ID f34f1b6800000025
2021-07-11T21:04:59.557004+00:00 sdc1001 kernel: [19011356.492319] PowerFlex mapVolIO_Do_CK:1447 :Mit: ffff8c3d42cc9400. Retrying IO Type READ. Failed comb: 433800a0234. offsetInComb 5017600. SizeInLB 2048. SDS_ID f34f1b6800000025. Comb Gen 1. Head Gen 181.
2021-07-11T21:04:59.557019+00:00 sdc1001 kernel: [19011356.510597] PowerFlex mapVolIO_Do_CK:1461 :Mit: ffff8c3d42cc9400. Vol ID 0xfa49a360000009f. Last fault Status IO_FAULT_NOT_PRI(12).Last error Status ERROR(1) Retry count (1) chan (5)

 – Die Verwendung von iperf zum Testen der SDC-zu-SDS-Verbindung zeigt große Durchsatzschwankungen. Ein guter iperf-Lauf zeigt einen stabilen und konsistenten Durchsatz:

# iperf3 -c 192.168.1.145
Connecting to host 192.168.1.145, port 5201
[ 4] local 10.1.3.98 port 50318 connected to 192.168.1.145 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.41 GBytes 20.7 Gbits/sec 254 1.43 MBytes  
[ 4] 1.00-2.00 sec 2.02 GBytes 17.3 Gbits/sec 38 1.20 MBytes  
[ 4] 2.00-3.00 sec 2.46 GBytes 21.2 Gbits/sec 0 1.40 MBytes  
[ 4] 3.00-4.00 sec 1.26 GBytes 10.8 Gbits/sec 18 1.42 MBytes  
[ 4] 4.00-5.00 sec 2.15 GBytes 18.4 Gbits/sec 17 1.22 MBytes  
[ 4] 5.00-6.00 sec 2.40 GBytes 20.6 Gbits/sec 2 1.00 MBytes  
[ 4] 6.00-7.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.28 MBytes  
[ 4] 7.00-8.00 sec 2.28 GBytes 19.6 Gbits/sec 5 1.44 MBytes  
[ 4] 8.00-9.00 sec 874 MBytes 7.33 Gbits/sec 234 1.46 MBytes  
[ 4] 9.00-10.00 sec 814 MBytes 6.83 Gbits/sec 2407 1.48 MBytes  

Auswirkungen

 - Keine optimale Anwendungsperformance auf den SDC-Hosts
: Mögliche I/O-Fehler bei starker Auslastung auf den SDC-Hosts 

In diesem Fall wurden die Betriebssystem-Standardeinstellungen der Kernel-Tuningparameter des SDC-Hosts geändert, was dazu führte, dass der TCP-Stack bei hohen Auslastungen nicht optimal funktionierte. 

Entfernen Sie die sysctl-Kernel-Tuningparameter und setzen Sie sie auf die Standardeinstellung zurück. Wenden Sie sich an den Betriebssystemanbieter, um zu erfahren, was die verschiedenen Kernel-Tuningparameter bewirken und wie sich mögliche Änderungen auf das Betriebssystem auswirken können, auf dem die SDCs ausgeführt werden. 

Betroffene Versionen

VxFlex OS 3.0.x
PowerFlex v3.5.x
PowerFlex v3.6.x

Behoben in Version

Nicht zutreffend, kein PowerFlex-Problem