PowerFlex 3.X: SDCs Disconnect From SDSs Under Heavy Read IO Load

Summary: SDCs disconnect from SDSs under heavy read IO load.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Certain SDCs, with a heavier read IO load, will disconnect briefly from SDSs. 

 

- Events as seen from the MDM events.txt:

2021-07-10 22:31:22.274 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 
2021-07-10 22:31:23.897 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.173 of SDS fs03sds058; ID: f34f1b7c00000039 
2021-07-10 23:14:42.041 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 
2021-07-10 23:14:44.041 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.112 of SDS fs01sds013; ID: f34f1b4f0000000c 
2021-07-10 23:21:13.722 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 
2021-07-10 23:21:15.561 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.134 of SDS fs02sds027; ID: f34f1b5d0000001a 
2021-07-10 23:31:09.561 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 
2021-07-10 23:31:11.353 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.147 of SDS fs02sds040; ID: f34f1b6a00000027 
2021-07-11 20:57:22.105 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 
2021-07-11 20:57:23.709 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.172 of SDS fs03sds057; ID: f34f1b7b00000038 
2021-07-11 21:02:20.985 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 
2021-07-11 21:02:21.996 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.181 of SDS fs03sds066; ID: f34f1b8400000041 
2021-07-11 21:04:58.553 SDC_DISCONNECTED_FROM_SDS_IP WARNING  	 SDC Name: sdc1001; ID: 33ceefc60000000e disconnected from the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 
2021-07-11 21:05:00.473 SDC_CONNECTED_TO_SDS_IP   INFO     	 SDC Name: sdc1001; ID: 33ceefc60000000e is now connected to the IP address 192.168.1.145 of SDS fs02sds038; ID: f34f1b6800000025 
 


 - Events as seen from the SDC side: 

2021-07-11T21:04:58.553170+00:00 sdc1001 kernel: [19011355.478437] PowerFlex netCon_IsKaNeeded:4321 :CON ffff8c3eb591b900 peer 192.168.1.145 didn't receive message for 30 iterations.  Marking as down
2021-07-11T21:04:58.552199+00:00 sdc1001 kernel: [19011355.491770] [23305989309] Disconnected from SDS with ID f34f1b6800000025
2021-07-11T21:04:59.156173+00:00 sdc1001 kernel: [19011355.598778] [23305989416] Connected to SDS with ID f34f1b6800000025
2021-07-11T21:04:59.557004+00:00 sdc1001 kernel: [19011356.492319] PowerFlex mapVolIO_Do_CK:1447 :Mit: ffff8c3d42cc9400. Retrying IO Type READ. Failed comb: 433800a0234. offsetInComb 5017600. SizeInLB 2048. SDS_ID f34f1b6800000025. Comb Gen 1. Head Gen 181.
2021-07-11T21:04:59.557019+00:00 sdc1001 kernel: [19011356.510597] PowerFlex mapVolIO_Do_CK:1461 :Mit: ffff8c3d42cc9400. Vol ID 0xfa49a360000009f. Last fault Status IO_FAULT_NOT_PRI(12).Last error Status ERROR(1) Retry count (1) chan (5)
 


 - Using iperf to test SDC to SDS connectivity shows large swings of throughput. A good iperf run shows steady and consistent throughput: 

# iperf3 -c 192.168.1.145
Connecting to host 192.168.1.145, port 5201
[ 4] local 10.1.3.98 port 50318 connected to 192.168.1.145 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.41 GBytes 20.7 Gbits/sec 254 1.43 MBytes  
[ 4] 1.00-2.00 sec 2.02 GBytes 17.3 Gbits/sec 38 1.20 MBytes  
[ 4] 2.00-3.00 sec 2.46 GBytes 21.2 Gbits/sec 0 1.40 MBytes  
[ 4] 3.00-4.00 sec 1.26 GBytes 10.8 Gbits/sec 18 1.42 MBytes  
[ 4] 4.00-5.00 sec 2.15 GBytes 18.4 Gbits/sec 17 1.22 MBytes  
[ 4] 5.00-6.00 sec 2.40 GBytes 20.6 Gbits/sec 2 1.00 MBytes  
[ 4] 6.00-7.00 sec 2.34 GBytes 20.1 Gbits/sec 0 1.28 MBytes  
[ 4] 7.00-8.00 sec 2.28 GBytes 19.6 Gbits/sec 5 1.44 MBytes  
[ 4] 8.00-9.00 sec 874 MBytes 7.33 Gbits/sec 234 1.46 MBytes  
[ 4] 9.00-10.00 sec 814 MBytes 6.83 Gbits/sec 2407 1.48 MBytes  
 

 

Impact

 - Less than optimal application performance on the SDC hosts
 - Possible IO errors under heavy load on the SDC hosts 

 

Cause

In this case, the SDC hosts kernel tuning parameters had been changed away from the OS defaults, which caused the TCP stack to not be able to work optimally under heavy loads. 

Resolution

Remove the sysctl kernel tuning parameters and set them back to default. Consult with the OS vendor on what the various kernel tuning parameters do and how possible changes can affect the OS the SDCs are running on. 

 

Impacted Versions

VxFlex OS 3.0.x
PowerFlex v3.5.x
PowerFlex v3.6.x

Fixed In Version

N/A, Not a PowerFlex issue

Affected Products

PowerFlex Software
Article Properties
Article Number: 000197487
Article Type: Solution
Last Modified: 13 May 2026
Version:  7
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.