PowerFlex 3.X: 읽기 IO 부하가 높을 때 SDC와 SDS의 연결이 끊어짐

읽기 IO 로드가 더 많은 특정 SDC는 SDS와의 연결이 잠시 끊깁니다. 

- MDM events.txt에 표시되는 이벤트:

2021-07-10 22:31:22.274 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 
2021-07-10 22:31:23.897 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 
2021-07-10 23:14:42.041 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 
2021-07-10 23:14:44.041 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 
2021-07-10 23:21:13.722 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 
2021-07-10 23:21:15.561 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 
2021-07-10 23:31:09.561 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 
2021-07-10 23:31:11.353 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 
2021-07-11 20:57:22.105 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 
2021-07-11 20:57:23.709 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 
2021-07-11 21:02:20.985 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 
2021-07-11 21:02:21.996 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 
2021-07-11 21:04:58.553 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 
2021-07-11 21:05:00.473 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 

 - SDC 측에 표시되는 이벤트:

2021-07-11T21:04:58.553170+00:00 sdc1001 kernel: [19011355.478437] PowerFlex netCon_IsKaNeeded:4321 :CON ffff8c3eb591b900 peer 192.168.1.145 didn't receive message for 30 iterations.  Marking as down
2021-07-11T21:04:58.552199+00:00 sdc1001 kernel: [19011355.491770] [23305989309] Disconnected from SDS with ID f34f1b6800000025
2021-07-11T21:04:59.156173+00:00 sdc1001 kernel: [19011355.598778] [23305989416] Connected to SDS with ID f34f1b6800000025
2021-07-11T21:04:59.557004+00:00 sdc1001 kernel: [19011356.492319] PowerFlex mapVolIO_Do_CK:1447 :Mit: ffff8c3d42cc9400. Retrying IO Type READ. Failed comb: 433800a0234. offsetInComb 5017600. SizeInLB 2048. SDS_ID f34f1b6800000025. Comb Gen 1. Head Gen 181.
2021-07-11T21:04:59.557019+00:00 sdc1001 kernel: [19011356.510597] PowerFlex mapVolIO_Do_CK:1461 :Mit: ffff8c3d42cc9400. Vol ID 0xfa49a360000009f. Last fault Status IO_FAULT_NOT_PRI(12).Last error Status ERROR(1) Retry count (1) chan (5)

 - iperf를 사용하여 SDC에서 SDS로의 연결을 테스트하면 처리량이 크게 변동됩니다. 양호한 iperf 실행은 안정적이고 일관된 처리량을 보여줍니다.

# iperf3 -c 192.168.1.145
Connecting to host 192.168.1.145, port 5201
[ 4] local 10.1.3.98 port 50318 connected to 192.168.1.145 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.41 GBytes 20.7 Gbits/sec 254 1.43 MBytes  
[ 4] 1.00-2.00 sec 2.02 GBytes 17.3 Gbits/sec 38 1.20 MBytes  
[ 4] 2.00-3.00 sec 2.46 GBytes 21.2 Gbits/sec 0 1.40 MBytes  
[ 4] 3.00-4.00 sec 1.26 GBytes 10.8 Gbits/sec 18 1.42 MBytes  
[ 4] 4.00-5.00 sec 2.15 GBytes 18.4 Gbits/sec 17 1.22 MBytes  
[ 4] 5.00-6.00 sec 2.40 GBytes 20.6 Gbits/sec 2 1.00 MBytes  
[ 4] 6.00-7.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.28 MBytes  
[ 4] 7.00-8.00 sec 2.28 GBytes 19.6 Gbits/sec 5 1.44 MBytes  
[ 4] 8.00-9.00 sec 874 MBytes 7.33 Gbits/sec 234 1.46 MBytes  
[ 4] 9.00-10.00 sec 814 MBytes 6.83 Gbits/sec 2407 1.48 MBytes  

영향

 - SDC 호스트에서 최적의 애플리케이션 성능보다 낮음
 - SDC 호스트에서 과부하 시 IO 오류가 발생할 수 있음 

이 경우 SDC 호스트 커널 튜닝 매개변수가 OS 기본값에서 변경되어 TCP 스택이 과부하 상태에서는 최적으로 작동할 수 없게 되었습니다. 

sysctl 커널 튜닝 매개변수를 제거하고 기본값으로 다시 설정합니다. 다양한 커널 튜닝 매개변수가 수행하는 작업과 가능한 변경 사항이 SDC가 실행되는 OS에 미칠 수 있는 영향에 대해서는 OS 공급업체에 문의하십시오. 

영향을 받는 버전

VxFlex OS 3.0.x
PowerFlex v3.5.x
PowerFlex v3.6.x

수정된 버전

해당 없음, PowerFlex 문제 아님, N/A, Not a PowerFlex issue