You can check this with the NST yourself to see if the ports need to be initialized. There is an oppurtunity to cancel before any action is taken (at the summary screen). If you do need to initialize and reboot, it would be a coordinated reboot similar to an NDU (SPB, wait 360 seconds, SPA).
We had a CX4-480 that EMC upgraded to a CX4-960 and we ran into exactly the same thing. When they added the additional IO modules they didn't initialize them. We had to take another 4 hour outage window to have EMC back in to initialize the ports properly. It required reboots of the SPs and I'm not sure what else. I wasn't directly involved in the change, just in getting EMC to fix the issue (they were supposed to do it all with only one outage).
Based on that, I'm not sure if it is something you can do yourself or if the CE has to come in to complete it. Have to check with the CE.
You think you are mad? EMC even had a Project Manager working to make sure our CX4-480 to 960 upgrade went smooth. We planned it for well over a month. We scheduled downtime on many, many production servers. We did everything right. Then when EMC came back a few days later to connect the additional DAEs they couldn't do it. We had to schedule another outage on the same production servers right before RRSP Season (which is not a good time to make changes in the IT infrastructure of a Financial Services company).
That had visibility all the way up to the CIOs and above.
Somehow the phrase "We are not amused" just didn't quite cover it!
On the bright side, the second four hour outage window did fix it for us
I was going to ask the same thing. Was it a rolling reboot of the SPs, thereby leaving the hosts online? Or was it a complete outage?
I am in the same situation, took months planning the upgrade, took months to schedule the downtime, and had a project manager assigned to cover all the details. Downtime is very hard to come by in our infrastructure. We have one six hour maintenance window each month to address any hardware upgrades, OS patching, upgrades, etc... that I have to share with all the other server admins in the data center. So I don't always have time alloted each month to upgrade PowerPath or install the latest HBA firmware and drivers. It could be months before I have a window to do any work. Now I have projects that are being deployed that are waiting on this storage and the next time I have a maintenance window is the end of May. Oh yes I do understand!
I just got off the phone with one of the engineers that performed the conversion from the Cx3-80 to the CX4-960. He said that he looked through the procedure for the conversion and there was not a section that mentioned initializing the extra backend ports. This may be why yours did not get initialized as well Allen.
He did say that it looks like it is a NDU reboot, and can be done through the NST or the CLI. I am waiting to hear back from support though to confirm this. If it is a rolling reboot, then that is good, I can just do that in the evening and won't need to take any hosts offline.
Thanks for the information and confirmation that it is just a coordinated reboot. That is something I can accommodate. Is it the Install I/O Module wizard in the NST?
I thought our SAM was al over escalating this, but I'll let him know it is still an issue as well. Maybe he can get some action on this as well. The more internal resources push this, the more likely they will get it resolved immediately.
I wish our environment was as easy to work with as that. If I could have an array with only Windows and ESX I would love it!
The NST can initialize the additional back end ports, use the Add IO Modules or SFPs, and the wizard will identify the unitialized ports and allow you to initialize them.
I will check with the CPG team and make sure this is included in future revisions. Apologies for the poor experience with your Data in place upgrade.
Actually it was an "NDU" reboot of the SPs, but we have a large number of AIX hosts attached to this particular array and due to some configuration disagreements between our Unix Admins and us, we have to shut down the AIX hosts for all NDUs. We are working on that, but it is still an issue for us.
I forgot to mention that this all happened because the procedure generated for the CE was missing this step. That was confirmed after the fact. I was not happy with this answer since the whole thing was reviewed multiple times by different EMC resources, but I guess they still haven;t fixed the procedure generator (even though they found this problem back in January with us).
Thanks for the update Allen. We only have ESX and Windows, so NDUs are actually non-disruptive for us.
I have asked our Account Executive and local support reps to push this back up to EMC so that the procedure generator can be updated with this step. This is a pretty big step to miss for those customers that are depending on those extra ports.
Hey, it's not your fault (unless you write the procedure generator that failed to provide the entire process *lol)
Mistakes happen, and right now we are more concerned with getting it fixed so no one else gets affected. I feel bad about not pushing harder to get this resolved knowing that AranH (and probably others) have been affected as well.
bertog
61 Posts
0
March 17th, 2010 08:00
You can check this with the NST yourself to see if the ports need to be initialized. There is an oppurtunity to cancel before any action is taken (at the summary screen). If you do need to initialize and reboot, it would be a coordinated reboot similar to an NDU (SPB, wait 360 seconds, SPA).
AranH1
2.2K Posts
0
March 16th, 2010 13:00
Now that makes me mad. I bet you are right. Well off to call support...
Allen Ward
4 Operator
•
2.1K Posts
1
March 16th, 2010 13:00
We had a CX4-480 that EMC upgraded to a CX4-960 and we ran into exactly the same thing. When they added the additional IO modules they didn't initialize them. We had to take another 4 hour outage window to have EMC back in to initialize the ports properly. It required reboots of the SPs and I'm not sure what else. I wasn't directly involved in the change, just in getting EMC to fix the issue (they were supposed to do it all with only one outage).
Based on that, I'm not sure if it is something you can do yourself or if the CE has to come in to complete it. Have to check with the CE.
AranH1
2.2K Posts
0
March 16th, 2010 13:00
...and here is how it looks from Navisphere:
Allen Ward
4 Operator
•
2.1K Posts
0
March 16th, 2010 13:00
You think you are mad? EMC even had a Project Manager working to make sure our CX4-480 to 960 upgrade went smooth. We planned it for well over a month. We scheduled downtime on many, many production servers. We did everything right. Then when EMC came back a few days later to connect the additional DAEs they couldn't do it. We had to schedule another outage on the same production servers right before RRSP Season (which is not a good time to make changes in the IT infrastructure of a Financial Services company).
That had visibility all the way up to the CIOs and above.
Somehow the phrase "We are not amused" just didn't quite cover it!
On the bright side, the second four hour outage window did fix it for us
AranH1
2.2K Posts
0
March 16th, 2010 13:00
I really hope the answer doesn't involve an SP reboot.... just have a bad feeling that it will
AranH1
2.2K Posts
0
March 16th, 2010 14:00
I was going to ask the same thing. Was it a rolling reboot of the SPs, thereby leaving the hosts online? Or was it a complete outage?
I am in the same situation, took months planning the upgrade, took months to schedule the downtime, and had a project manager assigned to cover all the details. Downtime is very hard to come by in our infrastructure. We have one six hour maintenance window each month to address any hardware upgrades, OS patching, upgrades, etc... that I have to share with all the other server admins in the data center. So I don't always have time alloted each month to upgrade PowerPath or install the latest HBA firmware and drivers. It could be months before I have a window to do any work. Now I have projects that are being deployed that are waiting on this storage and the next time I have a maintenance window is the end of May. Oh yes I do understand!
dynamox
9 Legend
•
20.4K Posts
0
March 16th, 2010 14:00
Allen ..i take it rolling reboot was not an option for that issue ?
AranH1
2.2K Posts
0
March 16th, 2010 15:00
I just got off the phone with one of the engineers that performed the conversion from the Cx3-80 to the CX4-960. He said that he looked through the procedure for the conversion and there was not a section that mentioned initializing the extra backend ports. This may be why yours did not get initialized as well Allen.
He did say that it looks like it is a NDU reboot, and can be done through the NST or the CLI. I am waiting to hear back from support though to confirm this. If it is a rolling reboot, then that is good, I can just do that in the evening and won't need to take any hosts offline.
AranH1
2.2K Posts
0
March 17th, 2010 08:00
Bertog,
Thanks for the information and confirmation that it is just a coordinated reboot. That is something I can accommodate. Is it the Install I/O Module wizard in the NST?
Allen Ward
4 Operator
•
2.1K Posts
0
March 17th, 2010 09:00
I thought our SAM was al over escalating this, but I'll let him know it is still an issue as well. Maybe he can get some action on this as well. The more internal resources push this, the more likely they will get it resolved immediately.
I wish our environment was as easy to work with as that. If I could have an array with only Windows and ESX I would love it!
bertog
61 Posts
0
March 17th, 2010 09:00
The NST can initialize the additional back end ports, use the Add IO Modules or SFPs, and the wizard will identify the unitialized ports and allow you to initialize them.
I will check with the CPG team and make sure this is included in future revisions. Apologies for the poor experience with your Data in place upgrade.
Allen Ward
4 Operator
•
2.1K Posts
0
March 17th, 2010 09:00
Actually it was an "NDU" reboot of the SPs, but we have a large number of AIX hosts attached to this particular array and due to some configuration disagreements between our Unix Admins and us, we have to shut down the AIX hosts for all NDUs. We are working on that, but it is still an issue for us.
I forgot to mention that this all happened because the procedure generated for the CE was missing this step. That was confirmed after the fact. I was not happy with this answer since the whole thing was reviewed multiple times by different EMC resources, but I guess they still haven;t fixed the procedure generator (even though they found this problem back in January with us).
AranH1
2.2K Posts
0
March 17th, 2010 09:00
Thanks for the update Allen. We only have ESX and Windows, so NDUs are actually non-disruptive for us.
I have asked our Account Executive and local support reps to push this back up to EMC so that the procedure generator can be updated with this step. This is a pretty big step to miss for those customers that are depending on those extra ports.
Allen Ward
4 Operator
•
2.1K Posts
0
March 17th, 2010 10:00
Hey, it's not your fault (unless you write the procedure generator that failed to provide the entire process *lol)
Mistakes happen, and right now we are more concerned with getting it fixed so no one else gets affected. I feel bad about not pushing harder to get this resolved knowing that AranH (and probably others) have been affected as well.