Ask the Expert: Reducing the Pain and Risk of IT Infrastructure Upgrades and Patches through Converged Systems and Software

Welcome to this Dell EMC Community Network Ask the Expert conversation. Our discussion is now open for interaction.

When formulating your question please consider the following:

Sign in or register to engage – You must login or, if you’re new, register to DECN in order to post a question, like, bookmark or follow this discussion. https://developer-content.emc.com/login/login.asp.

Be respectful – SMEs are volunteering to assist you and will provide you with information that they have access to.

Provide details and background – Our SMEs will try their best to provide a full answer for your satisfaction, but will appreciate as much information you can provide about your inquiry.

Keep focus and within topic – The expectation is to receive questions about the topic is that is being presented on the title and description of this event. We would appreciate if questions are kept within those boundaries.

Be appreciative – Many tools available for this, likes, user ratings, follow, bookmark and the good old “thank you”.

We’re delighted to know you’ve decided to join our debate and hope you receive the answers you’re eager to find. Thanks!

People newly exposed to Dell EMC (formerly VCE) converged platforms are intrigued by the release certification matrix (RCM). Perhaps Scott Redfern, Senior Director of Engineering, will shed some light on the commonly asked questions:  What is the RCM?  What kind of validation testing does Dell EMC do? How many tests and how many hours of testing?

I'd like ESG Lab to explain the results of their system upgrade and patch process testing in context of IT transformation goals that many organizations share.  How can the release certification matrix process help drive IT agility and more overall IT process efficiency?

Great question, @DavidWHayward

Organizations are always looking for ways to improve IT operational efficiency, whether it be related to deployment, management, protection, security etc. One key area where organizations focus on is ongoing maintenance – in other words, keeping your infrastructure up-to-date. This is important not just to ensure all resources are running optimally, but also to ensure everything is secure. All it takes is one out-of-date piece of hardware or an unpatched security vulnerability to bring an infrastructure to its knees. Traditionally, this process takes a very long time, especially in environments with disparate resources spread out across multiple sites. Management complexity, limited understanding due to lack of documentation, personnel buy-in, and cost are just some of the things that lead to extensive delays in updating components to the latest and greatest software. Imagine if you could easily audit your infrastructure in seconds to understand what needs updating, quickly download only the upgrades you want/need, and then apply those updates knowing they've already been vetted based on your exact infrastructure. The time savings and increase in IT agility is obvious.

The goal of our testing was to understand the process that goes into maintaining a fully-functioning IT infrastructure and quantify the time it takes to satisfy all of the requirements across all phases of the upgrade processes. For updating components in an infrastructure, it’s not just about stumbling upon a new rev of software and updating one piece of hardware on a whim. It’s about understanding what needs updating, what impact the update will have on the rest of the infrastructure, if downtime will be experienced, how long will the update take, and was the update successful. To that end, VCE’s release certification matrix takes all the guess work out of the update process, saving hours of time when updating an infrastructure – of course in this case the infrastructure is a VCE Vblock. They do all the research, testing, and validation of new updates for you. They even go a step further and guide you on how to complete the updates, including update order and expected timing. And it’s worth mentioning that with VCE, often times upgrades can be done without experiencing any downtime. The full report goes into much more detail about the steps in each phase of the update process and expected timing to complete each phase based on infrastructure type and component count. In short, the RCM plays an integral role in not only improving IT operational efficiency and agility, but enabling higher levels of IT administrator productivity by dramatically reducing the time required to maintain and upgrade a complete IT infrastructure.

Customers often ask us if they purchased a converged system  from Dell EMC, then how would the company help them deal with components' technical and security alerts and patches for remediation differently than if they purchased multi-vendor components separately and integrated them. Ted Balman, based on your experience with these issues in the field, will you explain with a couple of real-life examples?

That’s an excellent question Dave!  I’ll provide some color and ask Ted Balman to provide some added perspective as well.  Ted is Director of Core Technologies within Platform Engineering and two of the areas he is responsible for are Release Certification Matrices (RCM) and Security Threat Monitoring and Remediation (PSIRT) for our Blocks, Racks and Vscale products.

RCMs are an essential part of Dell EMC’s Blocks, Racks and Vscale value proposition.  RCMs are released 10 times per year with two being major releases and the other 8 being minor releases.  You might think that RCM is just a new name for an HCL or support matrix, but it’s far more.  Let me explain… a support matrix is a list of different components that are either known or believed to be compatible with one another.  Navigating a support matrix can be complex, and even with a front-end tool it requires knowing all of the components of your configuration to enter into the tool.  The resulting output is usually a collection of potential configurations, each with a series of release notes that must be read to see if additional drivers, BIOS updates, are required.  Then the customer pulls all of these together manually to end up with a valid update pack.

A Release Certification Matrix on the other hand has done all of that for you.  An RCM release is not a matrix, it is a bundle of every piece of software, driver and firmware/BIOS that is required for your system.  The version of each of those is carefully considered and mated for optimal interaction as a converged infrastructure product.  The end result is a bundle of software that can be downloaded and pre-staged on your Vblock, VxBlock, or VxRack by our Vision software and it encompasses the necessary changes for all supported components.  For example, RCM 6.0.9.1 identifies approximately 55 different components that can be included in a VxBlock 740 configuration and the certified software release required on each one in order to be compliant with that RCM. Certified means that we have tested a VxBlock 740 with all of those components installed, running the specified software, driver, firmware versions, and run hundreds to thousands of regression tests against it for 1-2 months to ensure that we can say it meets our quality, stability, and performance expectations for your workloads.

The last major RCM release encompassed over 3,500 individual tests spanning a period of about 6 weeks (2 agile sprints) in our labs before it was deemed ready for release.  Phew, that's a lot so it's a good thing our tests are nearly 100% automated!

Thanks Scott for providing such a complete answer!  I'll just add a couple of real world examples on what drives us to make changes to the RCM.

1. Security vulnerabilities are the number one driver to RCM change.  Depending on the severity of the issue, we determine when the changes have to be made.  If the flaw is severe enough, we'll push it into an addendum ASAP to protect our customer base.  In the last couple of years, we have managed out multiple high level security vulnerabities.  A few of note are Ghost, Heartbleed and glibc. So far in 2016 the PSIRT process has managed out 73 vulnerabilities found in our products.

2. Another driver is major change to a specific component that customers are clamoring for.  A good example would be for the latest version of hypervisor from VMware.

Hopefully this additional note provides some good understanding on the key drivers.  There are others as well, but these two are the biggest contributors. 

Great points Ted!  Sometimes people wonder why it takes longer for us to add that new component or release which everyone is waiting for.  It comes down to the value of the RCM as a bundle that's certified for all components in the Vblock, VxBlock or VxRack.  If we are certifying a new piece of software, and it is only listed on a server supplier's HCL for 5 out of 6 models that we offer in our product, then we will wait until we can work with them to get that 6th model onto their HCL and then do our RCM testing.  Now if that model isn't one that you care about as an end-user it may frustrate you, but it assures all of our customers that they don't have to do a stare and compare at one of those matrix eye charts to see if their component made it into our test.

Fantastic information Mike... I loved reading your white paper because it hit on many salient points of why we spend so much time on our RCMs.  We are always looking at ways that we can make both the certification process more valuable to customers, and to help make the software update implementation process less complex for IT administrators through implementation services, instruction documents, Vision integration, and future automation.  So we hope we'll get some questions and feedback in that area too during this "Ask the Expert" discussion to help us zero in on how we can take that next quantum step in making RCMs even better!

We've conducted RCM feedback sessions at EMC World, our Customer Advisory Board, the VUG at VMworld and with individual customers & partners this year.  Don't tell anyone, but we are collecting requirements now for a next generation RCM approach... hint, hint.

This Ask the Expert event has officially ended, but don't let that retract you from asking more questions. At this point our SMEs, after addressing all questions previosly posted, are still welcomed to answer and continue the discussion though not required. Here is where we ask our community members to chime in and assist other users if they're able to come up with accurate information. Additonally, you're welcome to post other questions unrelated to the topic of this ATE on the Converged Platforms community.

Many thanks to our SMEs who selflessly made themselves available to answer questions. We also appreciate our community members for taking part of the discussion and ask so many interesting questions.

ATE events are made for your benefit as members of ECN. If you’re interested in pitching a topic or Subject Matter Experts we would be interested in hearing it. To learn more on what it takes to start an event please visit our Ask the Expert Program Space on ECN.