ProfessorChaos1

updated

15 years ago

0

3424

December 30th, 2011 17:00

Boot issue after DART upgrade attempt

Happy (early) New Year. I'm new to EMC, but I've immersed myself in the material over the past few weeks. I've found many answers on these forums and I now find myself reaching out and hoping somebody can help me.

I have three Celerra NS20s with CLARiiON CX3-10 backend storage. To take advantage of newer features, I backed up all of the important data from one and used USM to perfom a DART upgrade from 5.x to 6.x (I don't have the exact version in front of me, but it was an approved direct upgrade path according to the compatability documentation. All health checks were good in the pre-upgrade check). During the upgrade, I ran into an issue that I posted a solution on in this thread. Basically, the system had to be manually restarted (reboot -f), but I made sure everything was okay in the log (/var/tmp/upgrade.log). After rebooting and starting USM again, it picked up where it left off and continued on its happy way. Eventually, it got to the step where it had to upgrade the Control Station Linux. Everything went fine from what I could tell, but eventually USM came back and stated that it was taking longer than expected to reboot the Control Station. I wasn't able to ping the IP at all, so I went to the server room and hooked up a monitor and keyboard to the Control Station to see what was going on. It was attempting to boot, but was stuck on loading initrd.img. I rebooted several times, but it hits this right after GRUB tries to boot. In addition to the package for USM, I had a copy of the DART software on DVD. I booted into this and tried to do a repair, but I'm having the same problem. Can I reload the Control Station Linux from scratch without losing my data? I'm not entirely clear if a fresh load will remove any data or what will need to be reconfigured. What other options do I have? All of my storage is still available, but I can't manage it.

These were set up a while ago by a tech, but they have been moved to new location in a secure government building, so getting a tech out isn't a viable option at this point.

TL;DR - Attempted a DART upgrade and now I'm stuck at boot trying to load initrd.img. A tech coming out is unlikely.

Any input/advise is greatly appreciated. Thank you in advance.

  • 1367

    0

    Posted January 11th, 2012 06:00

    Well, the upgrade has been sucessful. I believe this document decribes what I was experiencing. Additionally, the process took MUCH longer than the estimated time to upgrade the control station Linux. Since the CS was unresponsive and based off of what I was seeing, I thought the kernel was hosed. I did still have to intervene and manually reboot, so the upgrade process did have something wrong. After a little over an hour, I could finally reach the CS. When I resumed the upgrade, it tried upgrading the CS Linux again, but went by in just a few seconds. I wish EMC had more documentation available to customers for those willing to look into it. I understand why they want certain tasks to be performed by their techs, but sparse documentation is more dangerous if one has only part of the procedure. Thank you to those who provided inputs. I will try to reciprocate as best as I can.

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    1148

    0

    Posted December 30th, 2011 18:00

    bummer, don't have an air card or tethering on your phone ?

  • 1148

    0

    Posted December 30th, 2011 18:00

    Wireless (including phones) is a big no-no where I'm at . I'll be the first to admit that it makes it much more difficult for issues like this. What I think I'll do is to get as much detail as possible so that I can do a WebEx from my car or at home on my laptop. Also, the EMC chat still functions at work, so I'm going to reengage that avenue next week when I'm back in the office. In the meantime, I hope to continue the dialogue here and hope that somebody may have info on this. Thanks again dynamox.

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    1148

    0

    Posted December 30th, 2011 18:00

    If the box is under active support contract, open a ticket. Some things can be done via WebEx.

  • 1148

    0

    Posted December 30th, 2011 18:00

    I'll double check our support status (I think we're on the tail end of it), but a WebEx will also not work. I was in a chat with an EMC support rep for something else and we attempted a WebEx, but it wouldn't run on our work systems that have outside connectivity. They have it locked down to the bare minimum (secure location). I appreciate the feedback nontheless. Thanks dynamox!

    Since this seems to be related to the Linux portion, I'm hoping some Linux gurus can chime in as well. Also, if anybody knows what the results would be if I did a fresh install of the Control Station Linux.

  • 1148

    0

    Posted December 30th, 2011 19:00

    Well, since it is a long weekend for me, I've taken the time to do some more research. I found this thread where a user had to have the CS replaced and he inquired about his settings being kept with the new equipment. @dynamox, you replied to that thread and indicated that configurations such as file systems, CIFS servers, etc would remain intact. I realize that my situation is somewhat different and that I won't have a tech on hand, but can anybody provide guidance as to whether or not a reinstall of the control station linux from scratch will muck with any of those settings during the install/configuration stage?

    Also from that thread, it seems as though I can get running again and import my previous settings from the NASDB backup. Since I'm pretty certain we weren't copying that to another location, hopefully I will be able to at least access the drive and pull it off, but simply having the CS running again with my shares, etc still available is my goal.

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    1148

    1

    Posted December 30th, 2011 20:00

    i had a CS go bad on me, everything kept running ok (except for snapsure schedules did not run). We had EMC CE come out and replace the thing, i do not know what he did after he physically replaced it but nothing was lost (except for some scripts that i had on the control station)

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    1148

    1

    Posted December 31st, 2011 04:00

    In order to do that you need the procedure for control station replacement which saves/restores some config

    I would try to get it fixed with customer service before re-installing though

  • 1149

    0

    Posted January 3rd, 2012 08:00

    Well, I was planning on tackling this when I got in this morning, but I had some other things come up and I won't be back to work until tomorrow. I know from this thread (and all others) that going through customer service is my best route. I'm am still going to leverage all of my support options through them and see what I can find out. I'm also going to lean on some local guys with more EMC background than me to get their inputs as well.

    With that being said, I want to make sure I have a way forward if those fall through. @Rainer_EMC, you mentioned that there are procedures for replacing the control station. While I wouldn't be replacing it, I'm sure it would have information pertinent to my situation. I searched the support site, but I couldn't find anything. Does anybody know if there are instructions available to customers? As much as I'd love to have something like this, it doesn't seem likely that there are instructions for anybody outside of service techs and certain partners.

    I think that somebody with more Linux knowledge than me may be able to offer some suggestions based off of the situation. At this point, we're really looking at a Linux issue and the right commands will likely get me to a point where I can boot the kernel. Since I can get to the GRUB menu, I'm going to see if it's checking for the updated kernel or at least find what's causing it to hang up.

  • 1149

    0

    Posted January 3rd, 2012 09:00

    Also, since I'm already heaping it on , a quick follow up on an earlier question. I know the control station can be replaced without affecting my shares, etc, but I'm wondering if reinstalling the control station will overwrite anything if I push forward with that route as a last ditch. It give me the option to to a destructive install of the control station Linux, but I'm not sure if that "destructive" part is only the control station itself that will have to be reconfigured, or if it will clear out everything. I know the routine by now , so I will ask EMC support this same question. I'm just looking for guidance here as well.

    Thank you for bearing with me. I really do appreciate those of you taking your own time to help me with something out of your own generosity. I can only offer you my thanks and try to reciprocate by trying to help others on here.