8 Posts
0
5654
Dell PowerVault MD3400 - Recurring Virtual disk not on preferred path error
Greetings,
I'm hoping someone can assist us.
Background:
We have a Dell PowerVault 3400 with 6x 1.92 Tb SSD drives and two RAID Controller Modules. The RAID Controller Module Firmware is 08.25.14.60. It repeatedly will switch its status to "Storage Array Needs Attention". When clicking the link to see the details in the Recovery Guru, the summary states it is "Virtual Disk Not On Preferred Path".
I followed this thread https://www.dell.com/community/DELL-EMC-Storage-Forum/DELL-Power-vault-MD3420-Virtual-disk-not-on-preferred-path-error/td-p/5155934 and followed the advice presented there:
I've verified the controller firmware is up to date and updated the drive firmware.
Issue:
Below is the main issue I'm asking for assistance on...
I can clear the error temporarily by going to Storage | Virtual Disk | Advanced | Redistribute Virtual Disks and the error will clear.
The error will return within a couple hours. Here is an example from yesterday where I redistributed the disks (starting with event sequence number 3737 @2:14:31 PM and it returning with sequence number 3749 @4:06:33 PM)
Below I have the Event Log Entries I think have details that may be helpful. What other information can I provide to paint a clear picture of the issue?
Thanks in advance!
Jeff
Event Log Entries:
Date/Time: 11/5/19 2:14:31 PM
Sequence number: 3737
Event type: 210A
Event priority: Informational
Description: Cache not enabled
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 0
Logged by: RAID Controller Module in slot 0
Date/Time: 11/5/19 2:14:31 PM
Sequence number: 3738
Event type: 212C
Event priority: Informational
Description: Virtual disk write-back caching restored
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 0
Logged by: RAID Controller Module in slot 0
Date/Time: 11/5/19 2:14:31 PM
Sequence number: 3739
Event type: 104
Event priority: Informational
Description: Needs attention condition resolved
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 0
Logged by: RAID Controller Module in slot 0
Date/Time: 11/5/19 2:14:31 PM
Sequence number: 3740
Event type: 502A
Event priority: Informational
Description: Virtual disk ownership assigned
Event specific codes: 0/0/0
Event category: Command
Component type: Virtual Disk
Component location: Virtual Disk VD1
Logged by: RAID Controller Module in slot 0
Date/Time: 11/5/19 2:14:31 PM
Sequence number: 3742
Event type: 210A
Event priority: Informational
Description: Cache not enabled
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 1
Logged by: RAID Controller Module in slot 1
Date/Time: 11/5/19 2:14:31 PM
Sequence number: 3743
Event type: 212C
Event priority: Informational
Description: Virtual disk write-back caching restored
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 1
Logged by: RAID Controller Module in slot 1
Date/Time: 11/5/19 4:06:32 PM
Sequence number: 3744
Event type: 210A
Event priority: Informational
Description: Cache not enabled
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 1
Logged by: RAID Controller Module in slot 1
Date/Time: 11/5/19 4:06:32 PM
Sequence number: 3745
Event type: 212C
Event priority: Informational
Description: Virtual disk write-back caching restored
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 1
Logged by: RAID Controller Module in slot 1
Date/Time: 11/5/19 4:06:32 PM
Sequence number: 3746
Event type: 210A
Event priority: Informational
Description: Cache not enabled
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 0
Logged by: RAID Controller Module in slot 0
Date/Time: 11/5/19 4:06:33 PM
Sequence number: 3747
Event type: 212C
Event priority: Informational
Description: Virtual disk write-back caching restored
Event specific codes: 0/0/0
Event category: Internal
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 0
Logged by: RAID Controller Module in slot 0
Date/Time: 11/5/19 4:06:33 PM
Sequence number: 3748
Event type: 2044
Event priority: Informational
Description: Virtual disk I/O shipping implicit transfer
Event specific codes: 0/0/0
Event category: Internal
Component type: Virtual Disk
Component location: Virtual Disk VD1
Logged by: RAID Controller Module in slot 0
Date/Time: 11/5/19 4:11:32 PM
Sequence number: 3749
Event type: 4011
Event priority: Warning
Description: Virtual disk not on preferred path due to failover
Event specific codes: 0/0/0
Event category: Error
Component type: RAID Controller Module
Component location: Enclosure 0, Slot 1
Logged by: RAID Controller Module in slot 1
SMARTMD
8 Posts
0
January 3rd, 2020 05:00
I'm happy to report that the issue is now resolved. The issue was that the SAS Interface card in one of the two connected VMWare host servers had failed. To identify which one, if you look at my server connection image it was slot 6 of "Server 1".
To summarize for anyone who is running into this issue in the future -
1. check the cable connections - if those are ok...
2. swap cables around to see if you can identify a bad cable - if those are ok...
3. If you have a DRAC, login and check the storage interfaces. This is what identified the issue for me - in one of the servers (server 1), only 1 of the 2 storage interfaces was still showing, in our other server (server 2) both interfaces were showing.
In hindsight, I should have done step 3 above first so I wouldn't have had to contact the remote hands team to check cables when it could have been seen if I just looked in the right place.
Hope this helps anyone in the future - thanks again to the Dell team for your help in resolving this, we now have a GREEN status for this unit/system. Cheers!
SMARTMD
8 Posts
0
November 7th, 2019 08:00
To add some additional information, the 6 SSD drives are manufactured by TOSHIBA.
Also, we do not have any of the Premium features enabled. I don't believe we have a iops load that would require anything special.
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
November 7th, 2019 09:00
Hello Jeff,
How many connections from each controller do you have to each host? How many virtual disks do you have on your MD3400? How many virtual disks are owned by each controller? Are you using replication on your MD3400?
In most cases that I have seen when you redistribute the virtual disks back to there owner and they change again it is normally due to load on one controller is more than the other controller.
Please let us know if you have any other questions.
SMARTMD
8 Posts
0
November 7th, 2019 10:00
Thanks for the response.
Q:How many connections from each controller do you have to each host?
A: 2 - Below is the diagram we followed when cabling.
Q: How many virtual disks do you have on your MD3400?
A: There is 1 virtual disk in the array - all the available storage belongs to the virtual disk.
Q: How many virtual disks are owned by each controller?
A: I guess the answer is 1 since there is only 1. Is there a way I can better answer this question?
Q: Are you using replication on your MD3400?
A: No - I haven't explicitly set up any replication.
In reference to your comment about the ownership changing due to load, what is the best way I can share with you the load for each controller?
Thanks again for your time and attention,
Jeff
SMARTMD
8 Posts
0
November 11th, 2019 12:00
Hi there - we are still looking to resolve this issue. It sounds like we should be looking at the load on each of the controllers as that could shed some light on the problem. Is there a process/test I can run to trap the counts so I can post them here?
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
November 13th, 2019 15:00
Hello Jeff,
Can I get you to send me a support bundle so that we can review it? I will send you a private message so that you can send me the log.
Please let us know if you have any other questions.
SMARTMD
8 Posts
0
November 14th, 2019 15:00
Thank you for offering to evaluate the support bundle file - has been sent to the email you sent via PM.
SMARTMD
8 Posts
0
November 21st, 2019 06:00
Quick bump - I'm checking in to confirm that the support bundle was received.
Thanks in advance.
SMARTMD
8 Posts
0
December 4th, 2019 14:00
Bumping again - can I provide more/better details? I'd love to get this resolved.
burge1980
3 Posts
0
December 5th, 2019 07:00
I am having the same issue. Can you redistribute the virtual disks during production hours, will it affect anything? I apologize for the ignorance, I wasn't the original person that set this up.
DELL-Sam L
Moderator
Moderator
•
6.9K Posts
0
December 5th, 2019 12:00
Hello SmartMD,
After reviewing the logs, we are able to see that ESX Host 003 port 5d0946604f356b00 is missing its connection to storage. What you will need to do is to look and make sure that the cable is not lose or came out of the port. Also, will need to make sure that you are using RR & not MRU. Also which driver are you using for your HBA’s in your hosts?
Please let us know if you have any other questions.