Unsolved
This post is more than 5 years old
3 Posts
0
24915
March 12th, 2015 08:00
MD1000/MD3000 problems
Hi,
I have an MD3000 and a MD1000 connected together and am having lots of disk disconnects and controller errors. Looking through the logs using MDSM I cant make out what is causing the fault.
The MD3000 has been in for a few years and was expanded with the MD1000 about a month and a half ago. It ran fine until last week when the MD1000 appeared to disconnect briefly. This made the virtual disk on it fail and we lost our mailserver. Right after the event all disks and controllers were showing as optimal.
At some point both of the RAID controllers in the 3000, and the EMMs in the 1000 have failure entries in the log. Lots of failed/degraded path errors, etc. After last week I left all the drives out of the 1000 to see if the system would settle down, which it appeared to do. Yesterday afternoon I pushed all the drives back in and created a disk group, but not a virtual disk. LKate on yesterday and this morning I get a whole raft of errors and the 3000 appears to fall off the iSCSI network for tens of seconds.
Any ideas anyone?
Can I hotplug the modules in the 1000, one at a time to reseat them? I did take controller 1 on the 3000 offline and reseat it as I thouthg that was where the errors were coming from. Both PSU in the 3000 have been reseated too.
It's driving me mad!
0 events found
DELL-Sam L
Moderator
•
7.9K Posts
0
March 12th, 2015 12:00
Hello SCLltd,
What is the current version of firmware that is running on your MD3000? Off the top of my head I would say that it sounds like you may have a firmware issue or a setting on your MD1000 may not be correct. The only way to be sure is to gather a support bundle and review it to see what all the errors are & how to resolve them. I will send you a private message that you can reply back with the support bundle.
Please let us know if you have any other questions.
SCLltd
3 Posts
0
March 13th, 2015 02:00
Thanks very much Sam.
I've followed the instructions in your message and uploaded the support bundle for you look at.
Simon.
DELL-Sam L
Moderator
•
7.9K Posts
1
March 13th, 2015 08:00
Hello SCLltd,
Thanks for uploading the logs as that helps looking into your issue. So after looking at the logs I can see that the MD1000 is being seen by your MD3000i. I also can see that the MD1000 is showing that all the drives were present and then some showed as removed.
One other thing is that when you installed your MD1000 & connected it to your MD3000i did you connect it in the following order:
If you connected your MD1000 as listed above then what I would do is if you have a spare PERC5/e or 6/e that you can use to connect the MD1000 to so that we can ensure that the MD1000 will present all drives to the PERC card correctly. If the MD1000 shows up correctly when connected to the PERC card then what I would do is to upgrade the EMM to the latest version of firmware. Here is the link to the EMM firmware just incase you need it. http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=8M4NH&fileId=2731100091&osCode=WNET&productCode=powervault-md1000&languageCode=EN&categoryId=FW
If you are able to test the MD1000 and it shows as optimal then what I would do is to connect it back to the MD3000 as listed above & confirm that it is seen as it should will all enclosures and drives.
Once you are able to get the MD1000 seen by the MD3000i one other thing that I would do is to update the firmware on the MD3000i as well. Here are the steps to upgrade the firmware on your MD3000i & the firmware files as well.
MD3000i controller firmware
http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=R315164&fileId=2731121564&osCode=WNET&productCode=powervault-md3000i&languageCode=EN&categoryId=SI
HDD Firmware
http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=58K96&fileId=3387654475&osCode=WNET&productCode=powervault-md3000i&languageCode=EN&categoryId=EH
Resource CD
http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=R294936&fileId=2731116446&osCode=WNET&productCode=powervault-md3000i&languageCode=EN&categoryId=CD
Steps to update the firmware:
1. Ensure you have a tested backup of the data contained on the array
2. It is highly recommended to gather support bundle from the "Support" tab of the MD Storage Manager before performing any firmware upgrade operations. Also recommend getting another support bundle after the upgrade is complete.
3. You must stop all I/O going to the array
4. Load MDSM to verify connectivity, then close MDSM but leave the main management screen open
5. From the main management screen in MDSM use the “upgrade raid controller firmware” on the “devices” screen to upgrade to the latest firmware
6. Verify connectivity through MDSM
7. Update the hard drive firmware if applicable
8. Reboot the stack
a. Power down the server(s)
b.Power down the MD3000(i)
c. Power down any attached storage (MD1000/12xx)
d. Leave the power off for 2-3 minutes
e. Power up in reverse order
9. Verify connection to the array and virtual disks and verify that data is presented to each of the connected servers.
This update process will take 60-90 minutes but will vary depending on how long the reboots take to complete.
Please let us know if you have any other questions.
SCLltd
3 Posts
0
March 17th, 2015 10:00
Thanks Sam.
That connect/startup sequence is correct, that's how we did it. What bugs me is that it ran fine for a month or so before going wrong. We're going to disconnect the MD1000 and test it independently.
We are also considering not re-using the MD1000 with our MD3000 again as we have lost confidence in it. When it went wrong it caused big problems. We may build up a Freenas box using the MD1000 as the disk enclosure and just use it as storage for non essential stuff.
Thanks again,
Simon