UNSOLVED
High Latency On One Controller SCv3020
I'm setting up a new SCv3020 and am having a problem with high latency, but only on one of the controllers. This is running with a 3 host vSphere cluster and is the only attached storage, via iSCSI. There is a mixture of SSD and SAS drives but all the data is currently on the SSD drives.
So say I have volume 1 and volume 2 on the top controller, when cloning a VM to the same datastore I see high latency up to say 50ms and in some cases higher. Whilst with volume 3 and 4 on the bottom controller, performing the same operation the latency is sub 2ms.
If I swap the volumes between controllers, so 3 and 4 are on the top and 1 and 2 on the bottom - then the high latency remains on the top controller and the bottom controller is fine.
As far as I can tell everything is configured correctly and there are no warnings in Storage Center of vCenter.
Dell support thought it might be an issue with the cache on the top controller, so they advised us to pull it out for a few seconds so it would lose power and the cache would be cleared. We've done this but the issue remains.
When setting up the vSphere hosts there were some error messages when Storage Center was configuring everything - invalid Storage API request I think. But on running the configuration again it said they were already mapped and everything looked ok.
Anyone got any ideas?
I've looked in to this some more....
I've run load tests from within VM's and the performance is fine, throughput is good and latency under 2ms. The only time the high latency seems to be happening is when I'm cloning a VM from vCenter. I believe this should be using the SAN offload (Full Copy) so it suggests that is working with one controller but not the other? Any ideas what could cause that?
When the issue happens the high latency only shows at the controller and storage profile views in SC charting. The actual disks being written to all show sub 2ms latency. So again, seems something is up at the controller level?
We went through the configuration with Dell Support, whilst there were some issues but resolving them didn't help with the latency issue. Whilst they could see the latency issue, they couldn't see what was causing it so we ended up swapping out the entire controller which resolved the issue.
DELL-Joey C
Moderator
•
4170 Posts
•
20965 Points
1014
0
Posted March 15th, 2022 02:00
Hi @chenchen,
It seems you have posted new topic for you issue, I've commented on it.
DELL-Joey C
Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell
Did I answer your query? Please click on ‘Mark as Accepted Answer’ if I did.
DELL-Sam L
Community Manager
•
8058 Posts
•
34386 Points
3707
0
Posted August 1st, 2019 13:00
Hello jon1628,
Seeing as the issue stays on controller1 no matter which volumes are on that controller it would be best to have support use secure console into that controller so that we can watch when the operation is happening to see what is going on. I don’t think that there is an issue with your configuration, but it never can hurt to double check to make sure all is set as in the deployment guide for your SCv3020. https://topics-cdn.dell.com/pdf/storage-scv3000_deployment-guide_en-us.pdf
One other thing I would check before contacting support is that you are following Dell EMC SC Series Best Practices with VMware vSphere 5.x-6.x https://downloads.dell.com/manuals/all-products/esuprt_solutions_int/esuprt_solutions_int_solutions_resources/general-solution-resources_white-papers8_en-us.pdf
Please let us know if you have any other questions.