You made a very informative thread and I appreciate that. I do not disagree with your advice of not running a dual PERC configuration for a virtual environment, but one size does not fit all.
I want to make something very clear. The performance issue is not a hardware problem. There is nothing broken that can be fixed. The performance loss going to a dual shared PERC configuration is due to write cache being disabled. Write cache is disabled because the VRTX is not a full-featured storage appliance. Since it is not able to mirror cache across two controllers the caching is disabled to prevent data loss/corruption in the event of a controller failure.
Everyone does not have the same needs. We have options available that some customers do not understand. To them they do not make sense, but we have a very diverse customer base. If there is a feature available on our systems then it is there because we have customers that want it. Some customers may prefer the redundancy of dual PERCs over the performance of write caching. Everyone has different needs.
With all of that being said there is something we do need to address. It was mentioned that sales reps were selling this solution with false expectations. Customers have told us that their sales reps told them they would see no performance loss, and others have mentioned that they were told they would see a performance benefit from this solution. That is something we will address with our sales teams. We do not intentionally lie to our customers, but if our sales teams were not trained correctly on this product then we can remedy that.
I apologize for any inconvenience caused by improper expectations on this configuration. If you are dissatisfied with the dual PERC configuration then please reach out to your sales team and they will work with you to remedy it. Keep in mind that there is nothing that can be fixed because there is nothing wrong with the hardware. They may be able to work out a deal on something else or possibly return and refund the second PERC. There is not a blanket solution for everyone. The sales teams will work with you individually to find the best way to resolve this.
If you are dissatisfied with the resolution you are offered then let me know and I will see what I can do.
I had the same problem and I can't understand the phrase:
"Everyone does not have the same needs. We have options available that some customers do not understand. To them they do not make sense, but we have a very diverse customer base. If there is a feature available on our systems then it is there because we have customers that want it. Some customers may prefer the redundancy of dual PERCs over the performance of write caching. Everyone has different needs."
Absolutly nobody mentioned, write performace will breake down with dual controllers. You can't read it anywhere. Just put the following pictures to your datasheets and marketing material and your customer will know what to do. This picture tells you everything.
I studied nearly every document close to the VRTX-System during the last 30 days. I was calling the support team, I was calling the DELL sales representative. Nobody mentioned the poor performance with dual PERC8 at my first calls. So I was looking on my side. Maybe I made a mistake. My sales reprentative called me after 1,5 months because he wanted to know if I want to buy a secound one becaus he knew I needed an additional system. I told him: No. I don't buy a secound VRTX because the write performance is unacceptable and nobody can find a mistake. After this he told me that he can remember that he heard something about that and I should call the support once again. At first the support team told me: You already called us one month before. You have the same mistake again? I told him: Not again - still. I have it still. They told me once again: Send me the log, bla, bla. I said: We already did it and I don't do it again. After telling him that my sales representative told me that there is an issue I got the answer we look after it and after additional 1,5 weeks I sent a mail once again and I got the answer from the technican: My supervisor will call you back. The supervisor called me back the next they and he told me the truth.
DELL never mentioned the fact of poor write performance with dual PERC8 controllers before I bought it. They mentioned it after massive pressure. I think the sales representative didn't know it.
All I can say: We removed the 2nd Controller and solved the rest of the problem with the sales representative. Whether I trust DELL? We will see. I still have to buy the 2nd system.
I really agree on what you are saying, and to me the statement below is just a lousy, unsatisfactory and condescending excuse:
"Everyone does not have the same needs. We have options available that some customers do not understand. To them they do not make sense, but we have a very diverse customer base. If there is a feature available on our systems then it is there because we have customers that want it. Some customers may prefer the redundancy of dual PERCs over the performance of write caching. Everyone has different needs."
Furthermore they still also have failed to answer my question "So I am really wondering what kind of configuration actually is suitable for a dual PERC setup." And it is kind of a given that "Everyone has different needs."
It would be a lot more appreciated and trustworthy if they would just admit their faulty information/wrong judgment. To me this isn't really good publicity for Dell. But hey, "Everyone has different needs.".
I really hope more customers will benefit from my findings and I also hope that Dell will come up with something better.
Thank you for this thread! We have had the extra controller card since we purchased the VRTX in February 2014. I was planning to install it soon as I saw the firmware was released for it. I am glad I ran across this thread, we can't afford the performance hit on the VRTX.
I sent the secound controller back. DELL told me, there will be a software update at the end of this year or at the beginning of the next year.
I don't need it any more. For me it is a problem of time. I have to travel to insert this card once again 8 hours. I told them, I will keep it if they can send me a technican to insert it once again, but for me it is not realy possible. I don't have time for it. I already spent to much time on the hardware.
Thanks for your time, effort and sharing the detail in this forum. It is much appreciated.
Thought you might like to know that my Dell sales rep alerted me to this issue in between purchase and shipping. We had the option to go Single or Dual PERC. Thanks for the information provided here, as I am much better able to understand the issue. At least this is not a surprise nor a headache for us.
No doubt, your efforts have helped this to be a smoother experience for us and other Dell customers. (fingers crossed!)
It is no longer required to remove the second SPERC controller to be able to use write-back instead of write-through as described in Erik's post. You can now disable the second PERC8 controller from the CMC. In the current firmware (1.35), the second disabled controller can be set as disabled and requires manual intervention to activate if the primary active controller were to fail. The usefulness of this is not really detailed in the patch notes, but this "fix" is meant to allow people to enable Write-Back and get rid of the abysmal performance you have when using both controllers in Write-Through.
Automatic failover (cold failover, causing disconnect) is a feature that will be release later. A lot later will be the actual "active/active" firmware update, which would allow "live" failover without downtime.
Steps:
Download VRTX CMC firmware version 1.35 or higher.
Shut down all your blades.
In the CMC interface, press "Chassis Overview > Update".
Select both checkboxes for the CMC controllers in the "CMC firmware" heading and press "Apply CMC Update".
Enter the location of the CMC firmware file and Apply it.
The CMC will show you its progress. Uploading takes about 8 minutes, applying the update a few minutes more. The CMCs will reboot after applying the update and you will get kicked out of the webinterface.
Once the CMC has rebooted, browse to the "Storage > Controllers > Troubleshooting" section.
For the SPERC of your choice, select the "Disable RAID controller" option and apply. This will reboot your storage component.
Once rebooted, go to "Storage > Virtual Disks > Manage" and select "Edit: Write Policy" and pick "Write Back" instead of "Write Through" for all your virtual disks (unless you have a reason not to).
The change above will be carried out immediately, but it is still suggested to reset the CMC once more under "Chassis Overview > Power > Control" with "Reset CMC (warm boot)".
Boot your shut down blades.
This will allow you to have a second PERC8 installed in your VRTX, in case the other fails. But you will have to do a manual intervention to fail over in case of failure. I suppose this is primarily meant for hard to reach locations (remote offices without IT staff or easy access for Dell support technicians). This is also what we use it for.
Hopefully by end of the year we will have the automatic failover feature and then in the course of next year the true active/active config with write-back enabled (synchronized caches). I'm not going to hold my breath for the synchronized cache firmware fix... I suspect that won't be an easy one for Dell.
My post is the answer to your inquiry. The reason you are experiencing poor performance is because write caching is disabled in the dual PERC configuration. The posts you keep marking as the answer propose that there is a hardware fault. There is not a hardware issue with the controller. This issue is with understanding how cache and RAID function.
This configuration was released with redundancy as a focus. Many people purchased this with the expectation of performance as well. We are PLANNING to update the firmware to allow caching to work with a dual PERC configuration. There is no guarantee that it will be implemented or that it will allow the full performance of a single PERC. Until it has been released we are not going to set the expectation that it will allow a dual PERC configuration to perform as well as a single PERC configuration.
Thanks for your additions. Unfortunately for me it is not completely clear what you mean with caching enabled and or disabled. From what I understand if you set the write policy to "Write through" this is equal to having caching disabled. So I don't get how you enabled caching while using the "write through" policy.
Which caching did you enable/disable? In the VRTX Chassis management? In Windows? Both?
What are you running on the VRTX? Windows Servers with shared storage? A HyperV Failover Cluster?
Although you results seem to be better than my thorough testing, you would be surprised how much performance you gain when you use "write back" as caching policy. although to enable "write back" you have to disable the 2nd PERC controller and reconfigure the Virtual Disk to use "write back" as caching policy.
Daniel My
10 Elder
•
6.2K Posts
1
June 13th, 2014 10:00
Erik
You made a very informative thread and I appreciate that. I do not disagree with your advice of not running a dual PERC configuration for a virtual environment, but one size does not fit all.
I want to make something very clear. The performance issue is not a hardware problem. There is nothing broken that can be fixed. The performance loss going to a dual shared PERC configuration is due to write cache being disabled. Write cache is disabled because the VRTX is not a full-featured storage appliance. Since it is not able to mirror cache across two controllers the caching is disabled to prevent data loss/corruption in the event of a controller failure.
Everyone does not have the same needs. We have options available that some customers do not understand. To them they do not make sense, but we have a very diverse customer base. If there is a feature available on our systems then it is there because we have customers that want it. Some customers may prefer the redundancy of dual PERCs over the performance of write caching. Everyone has different needs.
With all of that being said there is something we do need to address. It was mentioned that sales reps were selling this solution with false expectations. Customers have told us that their sales reps told them they would see no performance loss, and others have mentioned that they were told they would see a performance benefit from this solution. That is something we will address with our sales teams. We do not intentionally lie to our customers, but if our sales teams were not trained correctly on this product then we can remedy that.
I apologize for any inconvenience caused by improper expectations on this configuration. If you are dissatisfied with the dual PERC configuration then please reach out to your sales team and they will work with you to remedy it. Keep in mind that there is nothing that can be fixed because there is nothing wrong with the hardware. They may be able to work out a deal on something else or possibly return and refund the second PERC. There is not a blanket solution for everyone. The sales teams will work with you individually to find the best way to resolve this.
If you are dissatisfied with the resolution you are offered then let me know and I will see what I can do.
Thanks
Erik Nettekoven
21 Posts
0
June 15th, 2014 14:00
Daniel,
Thank you for this thorough clarification. It is good to know where we are at. We will work something out with our local Dell sales representatives.
I would also like to thank you for your patience and understanding.
Regards,
Erik
Michael kainer
4 Posts
0
July 3rd, 2014 08:00
Hi!
I had the same problem and I can't understand the phrase:
"Everyone does not have the same needs. We have options available that some customers do not understand. To them they do not make sense, but we have a very diverse customer base. If there is a feature available on our systems then it is there because we have customers that want it. Some customers may prefer the redundancy of dual PERCs over the performance of write caching. Everyone has different needs."
Absolutly nobody mentioned, write performace will breake down with dual controllers. You can't read it anywhere. Just put the following pictures to your datasheets and marketing material and your customer will know what to do. This picture tells you everything.
I studied nearly every document close to the VRTX-System during the last 30 days. I was calling the support team, I was calling the DELL sales representative. Nobody mentioned the poor performance with dual PERC8 at my first calls. So I was looking on my side. Maybe I made a mistake. My sales reprentative called me after 1,5 months because he wanted to know if I want to buy a secound one becaus he knew I needed an additional system. I told him: No. I don't buy a secound VRTX because the write performance is unacceptable and nobody can find a mistake. After this he told me that he can remember that he heard something about that and I should call the support once again. At first the support team told me: You already called us one month before. You have the same mistake again? I told him: Not again - still. I have it still. They told me once again: Send me the log, bla, bla. I said: We already did it and I don't do it again. After telling him that my sales representative told me that there is an issue I got the answer we look after it and after additional 1,5 weeks I sent a mail once again and I got the answer from the technican: My supervisor will call you back. The supervisor called me back the next they and he told me the truth.
DELL never mentioned the fact of poor write performance with dual PERC8 controllers before I bought it. They mentioned it after massive pressure. I think the sales representative didn't know it.
All I can say: We removed the 2nd Controller and solved the rest of the problem with the sales representative. Whether I trust DELL? We will see. I still have to buy the 2nd system.
Regards
Michael
Erik Nettekoven
21 Posts
1
July 7th, 2014 03:00
Hi Michael,
I really agree on what you are saying, and to me the statement below is just a lousy, unsatisfactory and condescending excuse:
Furthermore they still also have failed to answer my question "So I am really wondering what kind of configuration actually is suitable for a dual PERC setup." And it is kind of a given that "Everyone has different needs."
It would be a lot more appreciated and trustworthy if they would just admit their faulty information/wrong judgment. To me this isn't really good publicity for Dell. But hey, "Everyone has different needs.".
I really hope more customers will benefit from my findings and I also hope that Dell will come up with something better.
krishart1616
1 Message
1
July 18th, 2014 13:00
Thank you for this thread! We have had the extra controller card since we purchased the VRTX in February 2014. I was planning to install it soon as I saw the firmware was released for it. I am glad I ran across this thread, we can't afford the performance hit on the VRTX.
Michael kainer
4 Posts
1
July 21st, 2014 00:00
Hi,
I sent the secound controller back. DELL told me, there will be a software update at the end of this year or at the beginning of the next year.
I don't need it any more. For me it is a problem of time. I have to travel to insert this card once again 8 hours. I told them, I will keep it if they can send me a technican to insert it once again, but for me it is not realy possible. I don't have time for it. I already spent to much time on the hardware.
davestagg
1 Message
1
July 31st, 2014 01:00
Erik,
Thanks for your time, effort and sharing the detail in this forum. It is much appreciated.
Thought you might like to know that my Dell sales rep alerted me to this issue in between purchase and shipping. We had the option to go Single or Dual PERC. Thanks for the information provided here, as I am much better able to understand the issue. At least this is not a surprise nor a headache for us.
No doubt, your efforts have helped this to be a smoother experience for us and other Dell customers. (fingers crossed!)
Dave.
Richard_artes
1 Rookie
•
35 Posts
1
July 31st, 2014 08:00
Dell have now stopped selling the VRTX with dual PERC controller. My order was cancelled and re-issued with a single PERC controller.
Godverdomme
2 Posts
1
August 6th, 2014 06:00
It is no longer required to remove the second SPERC controller to be able to use write-back instead of write-through as described in Erik's post. You can now disable the second PERC8 controller from the CMC. In the current firmware (1.35), the second disabled controller can be set as disabled and requires manual intervention to activate if the primary active controller were to fail. The usefulness of this is not really detailed in the patch notes, but this "fix" is meant to allow people to enable Write-Back and get rid of the abysmal performance you have when using both controllers in Write-Through.
Automatic failover (cold failover, causing disconnect) is a feature that will be release later. A lot later will be the actual "active/active" firmware update, which would allow "live" failover without downtime.
Steps:
This will allow you to have a second PERC8 installed in your VRTX, in case the other fails. But you will have to do a manual intervention to fail over in case of failure. I suppose this is primarily meant for hard to reach locations (remote offices without IT staff or easy access for Dell support technicians). This is also what we use it for.
Hopefully by end of the year we will have the automatic failover feature and then in the course of next year the true active/active config with write-back enabled (synchronized caches). I'm not going to hold my breath for the synchronized cache firmware fix... I suspect that won't be an easy one for Dell.
Erik Nettekoven
21 Posts
0
August 28th, 2014 01:00
Could you please stop marking the post "Posted byDELL-Daniel My on 13 Jun 2014 11:48 AM" as answer? Because it is not the answer to the issue.
Daniel My
10 Elder
•
6.2K Posts
0
August 30th, 2014 10:00
Erik
My post is the answer to your inquiry. The reason you are experiencing poor performance is because write caching is disabled in the dual PERC configuration. The posts you keep marking as the answer propose that there is a hardware fault. There is not a hardware issue with the controller. This issue is with understanding how cache and RAID function.
This configuration was released with redundancy as a focus. Many people purchased this with the expectation of performance as well. We are PLANNING to update the firmware to allow caching to work with a dual PERC configuration. There is no guarantee that it will be implemented or that it will allow the full performance of a single PERC. Until it has been released we are not going to set the expectation that it will allow a dual PERC configuration to perform as well as a single PERC configuration.
Thanks
Erik Nettekoven
21 Posts
0
September 3rd, 2014 08:00
Hi TTOONKA,
Thanks for your additions. Unfortunately for me it is not completely clear what you mean with caching enabled and or disabled. From what I understand if you set the write policy to "Write through" this is equal to having caching disabled. So I don't get how you enabled caching while using the "write through" policy.
Which caching did you enable/disable? In the VRTX Chassis management? In Windows? Both?
What are you running on the VRTX? Windows Servers with shared storage? A HyperV Failover Cluster?
Although you results seem to be better than my thorough testing, you would be surprised how much performance you gain when you use "write back" as caching policy. although to enable "write back" you have to disable the 2nd PERC controller and reconfigure the Virtual Disk to use "write back" as caching policy.
ttoonka
6 Posts
0
September 3rd, 2014 08:00
ttoonka
6 Posts
0
September 3rd, 2014 08:00
Thank you here for great discussion, I would like to publish my result.
I am running VRTX with PERC8 Shared in Active/Passive config with Write Through policy, disk cache is enabled.
RAID 10 from 8x1TB SAS HDD
PERC8 firmware version is 23.8.10-0059
My results are not that bad as you have published above.
ttoonka
6 Posts
0
September 3rd, 2014 08:00
the same test with Disk Chache disabled, so disk chache make big difference and it could be solution, but it increase a risk.