6 Posts
0
2409
DD 990 - DD FS wont start
We currently have a DD 990 system running DD OS 6.1.2.70-654892.
We have an issue where the FS wont restart.
We noticed following errors.
Fri Dec 31 15:26:34 2021 CRITICAL Environment Enclosure=1 EVT-ENVIRONMENT-00035: Critical failure caused an unexpected system shut down
Fri Dec 31 15:32:38 2021 ALERT Filesystem EVT-FILESYS-00010: DDFS process died after reboot; restarting.
Fri Dec 31 15:37:42 2021 ALERT Filesystem EVT-FILESYS-00011: DDFS process died; restarting.
Also see following messages in logs.
Jan 1 11:38:08 localhost /usr/sbin/irqbalance: WARNING: MSI interrupts found in /proc/interrupts.
Jan 1 11:45:46 st-dd1 ddfs[12942]: ERROR: MSG-INTRNL-00001: PANIC: ddr/segstore/ss_nvram.c: ssnv_restore: 1025: Fatal Error.
Jan 1 11:49:03 st-dd1 ddfs[20002]: ERROR: MSG-INTRNL-00001: PANIC: ddr/segstore/ss_nvram.c: ssnv_restore: 1025: Fatal Error.
Jan 1 11:52:33 st-dd1 ddfs[21560]: ERROR: MSG-INTRNL-00001: PANIC: ddr/segstore/ss_nvram.c: ssnv_restore: 1025: Fatal Error.
Jan 1 11:56:02 st-dd1 ddfs[23915]: ERROR: MSG-INTRNL-00001: PANIC: ddr/segstore/ss_nvram.c: ssnv_restore: 1025: Fatal Error.
Dec 31 15:18:59 localhost /usr/sbin/irqbalance: WARNING: MSI interrupts found in /proc/interrupts.
Already attempted reboots multiple times but seeing the same issue where the FS wont restart.
Any input on the fix would be greatly appreciated.
Thanks,
barry_beckers
393 Posts
0
January 4th, 2022 09:00
in the most recent "Data Domain: DD OS Software Versions" it states:
"DDOS 6.2:
DD160, DD620, DD990, DD2200, DD2500, DD3300, DD4200, DD4500, DD6300, DD6800, DD7200, DD9300, DD9500, DD9800 & DDVE
DDOS 7.2 & Above :
DD3300, DD4200, DD4500, DD6300, DD6800, DD6900, DD7200, DD9300, DD9400, DD9500, DD9800, DD9900 & DDVE"
So officially only a dd990 would support ddos6.2.
the dd890 and dd860's have an EOSS from March 31 2019 (but then again they are both from 2011 or so...). You might use the install precheck option when you uploaded the ddos 6.2 rpm, I assume it might object to being installed on the DD's in question if and when it performs a hardware type check.
I can't directly find it would support the latest and greatest 6.2 but KB articles from 2019 state circumstantial evidence as the dd89- and 860 are mentioned at the bottom while dealing with the "Data Domain: DD OS Software Versions" in which it talks about ddos 6.2.0 up until 6.2.0.30 at the time. so who knows it might accept that later version as well. If not then very likely still 6.2.0.30 (with all its bugs and issues that is).
barry_beckers
393 Posts
0
January 3rd, 2022 01:00
As you ask the question here, I imagine the dd990 is no longer under maintenance?
A dd990 is already out of standard support, so would require extended support contract?
Regardless of that, is there also any specific reason to still run an unsupported ddos 6.1 version? Even though a dd990 cannot be updated to ddos7.x, it can still run up until the latest and greatest 6.2, which might deal with the issue you are affected by, especially as it is not working at the moment? This assumes you still have access to the Dell support site to download a recent ddos 6.2 rpm? 6.2.1.40 or above is the target release for ddos 6.2, 6.2.1.70 being the latest.
But it is a dd not under contract, this would be a gamble, if not looking further what might be the cause? Then again Dell might still not even help even if the hardware is under contract until the ddos version is also within supported range again?
mahen03
6 Posts
0
January 4th, 2022 01:00
Thats correct, we no longer have support contract with dell.
Does DD990 support 6.2.x? I was under the impression that it cannot go beyond 6.1.x.
Currently the FS wont restart even after multiple reboots and there is no hardware issue on the system. Will we be able to upgrade to 6.2.x even though the FS is not working?
mahen03
6 Posts
0
January 4th, 2022 08:00
To add, the reason why I ask if these models are supported is due to it not being listed as supported systems in the release notes for 6.2.x
barry_beckers
393 Posts
0
January 4th, 2022 08:00
we have ddos 6.2.1.50 running on all of our dd990's (and pretty much all other systems, especially the ones that cannot be updated to 7.x), so that is a definite yes. might just solve your issue, can't be worse than it is now, I guess?
however it requires an Dell support site user account to be able to download that ddos 6.2 rpm.
I am not sure however if the install pre-check would check if the FS is down and would not continue if it wasn't or couldn't be started? as normally it would be enabled.
EDIT: I can't recall if your current version still enables you to use a special sequence to escape to the BASH sell on the DD, that was changed from a certain ddos 6.1 version onwards and would require escalating to Dell to get a token to be able to have them go to the BASH shell.
As you would not be able to do that without support, you would end up with a more closed down DD...
mahen03
6 Posts
0
January 4th, 2022 08:00
We have DD990, DD890 and DD860 in our environment. So would that be ok to upgrade all of these models?
I ran the pre-check with 6.2.x image and it ran fine with no errors but will the upgrade go through with the FS down, I am not sure. I am checking with my team to see how comfortable they are with upgrade. We are also trying to reseat the NVRAM and see if it helps since we saw some NVRAM messages in the logs.
ddfs[12942]: ERROR: MSG-INTRNL-00001: PANIC: ddr/segstore/ss_nvram.c: ssnv_restore: 1025: Fatal Error.
mahen03
6 Posts
0
January 4th, 2022 12:00
When you say Bash shell are you referring to SE mode?
barry_beckers
393 Posts
0
January 4th, 2022 13:00
SE mode you go into with the priv sets e command using the serialnumber to access it.
The bash shell uses a special sequence of commands to enter into the bash shell mode, as it has linux under the hood. Used often also by Dell engineers.
So if you never used it or are unaware of it, you would not even miss it (I used it in the past to use pam_tally to unlock a locked account, while in more recent ddos versions it will only temporary lock accounts).
mahen03
6 Posts
0
January 7th, 2022 22:00
Thanks for clarifying. We have never used the bash shell.
To update you, we were able to bring the FS by replacing the NVRAM although it did not show any hardware errors. We were able to upgrade 2 of our systems to 6.2.1.70 version to be at the latest supported versions. .