You're probably not going to be successful on your own. If the data is as important as you indicate, you should open a Time-and-Materials support case with EMC. For a price, they can get the Data Domain operational. Looking at the messages above, it's possible you have an error such as multiple drive failures or an NVRAM issue that's preventing the filesystem from coming up. Without extensive analysis of the logs, there's no way to know for sure.
You're NVRAM indicates it's OK but it was unrecoverable upon first startup during 2nd September - probably because it was powered off at the plug (not gracefully) and there was data in the NVRAM card and so the data was lost when the battery discharged.
You may still have NVRAM card issues but the backup data should be intact if you really need to restore, but as Karl says - you will need some support from Dell EMC on T&M to fix it.
You should at this point liaise with your Dell EMC account team to get this help.
Whilst Dell EMC can make no guarantees, I have no doubt that support can get the system fully functioning, perhaps/likely without parts.
So in theory, we should be able to fix this remotely.
In my experience, it is extremely rare (practically never) that we can't recover from most situations - I've seen many full recoveries even from multiple disk failures and broken raid groups. How long it takes is normally the only variable.
Since you do not have failed disks, you should be fine here with one of our filesystem gurus.
If you fs is disable, I would first ssh to it and check logs to see why it got disabled in the first place. Based on that, I would looks for next steps to fix it.
I can't say what is inside article as I get You Are Not Authorized to View The Content
It looks like as if it fell apart. You can try to enable it ofc, but if you see more cores created or if it disables itself again, you will have to go for support ticket. Heavily guessing, I would say that index db of fs needs to be rebuild, but I'm not sure about steps here as those are normally not performed by enduser. If you can wipe it out and start from scratch, then just do that (that is if no data of value is on it).
So from the above posts we can see that DDFS is failing to enable on this system due to the following error:
ddfs[28708]: ERROR: MSG-INTRNL-00001: PANIC: ddr/segstore/ss_index_mbuf.c: ssim_merge_full: 10718: Fatal Error: failed to read full index lvl 1 bucket 9336942.
This indicates that there are issues with on disk index files (in that at least part of the index file cannot be read). This is a fatal error and you will not be able to recover this system on your own without probably a reasonably lengthy engagement with support. Support should certainly be able to recover the system however without knowing more details its very hard to say whether recovery can be completed without data loss. If, however, something has physically damaged data on disk it is possible that not everything on the DDR will be able to be recovered.
You'll almost certainly need a time-and-materials case to have the logs analyzed. If it's an old version of DDOS, you might have hit a bug that's since been resolved. Or, it could be related to older firmware and the OS could be unable to cope with a particular condition when over 90% full. Either way, someone would need to research the root cause to give you more specifics.
jbrooksuk
208 Posts
0
September 7th, 2016 09:00
You probably have some faulty hardware, disks or NVRAM.
check;
alerts show current (list faults)
system show nvram (check condition of Nvram card)
disk show state (look for faulty disks "F")
df (check that ddvar is not full)
Regards,
Jonathan
umichklewis
3 Apprentice
•
1.2K Posts
0
September 7th, 2016 09:00
You're probably not going to be successful on your own. If the data is as important as you indicate, you should open a Time-and-Materials support case with EMC. For a price, they can get the Data Domain operational. Looking at the messages above, it's possible you have an error such as multiple drive failures or an NVRAM issue that's preventing the filesystem from coming up. Without extensive analysis of the logs, there's no way to know for sure.
Let us know if that helps!
Karl
jbrooksuk
208 Posts
0
September 7th, 2016 10:00
I agree with this.
Your disks/DIMMS look OK.
No other obvious hardware faults.
You're NVRAM indicates it's OK but it was unrecoverable upon first startup during 2nd September - probably because it was powered off at the plug (not gracefully) and there was data in the NVRAM card and so the data was lost when the battery discharged.
You may still have NVRAM card issues but the backup data should be intact if you really need to restore, but as Karl says - you will need some support from Dell EMC on T&M to fix it.
You should at this point liaise with your Dell EMC account team to get this help.
Hope that helps.
Jonathan
Kheldar9
1 Rookie
•
4 Posts
0
September 7th, 2016 12:00
Thank you for the quick replies.
Johnathan, I did the checks you suggested, and all hardware seems to be in order:
I will engage the data owner as find out if they are willing to spend money recovering the data.
Does EMC usually have a good/bad success rate in recoveries?
Sias
jbrooksuk
208 Posts
0
September 8th, 2016 00:00
Hello Sias,
Whilst Dell EMC can make no guarantees, I have no doubt that support can get the system fully functioning, perhaps/likely without parts.
So in theory, we should be able to fix this remotely.
In my experience, it is extremely rare (practically never) that we can't recover from most situations - I've seen many full recoveries even from multiple disk failures and broken raid groups. How long it takes is normally the only variable.
Since you do not have failed disks, you should be fine here with one of our filesystem gurus.
Let us know how you get on.
Regards, Jonathan
ble1
4 Operator
•
14.4K Posts
0
March 10th, 2017 12:00
If you fs is disable, I would first ssh to it and check logs to see why it got disabled in the first place. Based on that, I would looks for next steps to fix it.
I can't say what is inside article as I get You Are Not Authorized to View The Content
ble1
4 Operator
•
14.4K Posts
0
March 13th, 2017 07:00
It looks like as if it fell apart. You can try to enable it ofc, but if you see more cores created or if it disables itself again, you will have to go for support ticket. Heavily guessing, I would say that index db of fs needs to be rebuild, but I'm not sure about steps here as those are normally not performed by enduser. If you can wipe it out and start from scratch, then just do that (that is if no data of value is on it).
James_Ford
30 Posts
0
March 16th, 2017 03:00
So from the above posts we can see that DDFS is failing to enable on this system due to the following error:
ddfs[28708]: ERROR: MSG-INTRNL-00001: PANIC: ddr/segstore/ss_index_mbuf.c: ssim_merge_full: 10718: Fatal Error: failed to read full index lvl 1 bucket 9336942.
This indicates that there are issues with on disk index files (in that at least part of the index file cannot be read). This is a fatal error and you will not be able to recover this system on your own without probably a reasonably lengthy engagement with support. Support should certainly be able to recover the system however without knowing more details its very hard to say whether recovery can be completed without data loss. If, however, something has physically damaged data on disk it is possible that not everything on the DDR will be able to be recovered.
Thanks, James
umichklewis
3 Apprentice
•
1.2K Posts
0
March 17th, 2017 13:00
You'll almost certainly need a time-and-materials case to have the logs analyzed. If it's an old version of DDOS, you might have hit a bug that's since been resolved. Or, it could be related to older firmware and the OS could be unable to cope with a particular condition when over 90% full. Either way, someone would need to research the root cause to give you more specifics.
Let us know if that helps!
Karl