MD3000i and Modular Disk Storage Manager

Question

Howdy Folks,   We have a brand new MD3000i iSCSI array.  There are 4 servers connected to the array, via iSCSI, on a separate Cisco 3750 switch.  The array management interfaces are on our production network.  The configuration has been pretty simple, but I keep running into problems with the Modular Disk Storage Manager (MDSM).   After using the MDSM for a few days, it looses the connection to the storage modules on the MD3000i.  I can still ping the management and iSCSI interfaces of the array and iSCSI still works flawlessly from the servers, but I can't manage the array at all.   The MDSM says the storage array is currently unresponsive.  Removing the array and trying to add it back doesn't work.  I have to physically power down the array to start managing it again.   I have checked all the ports for duplex/speed mis-matches on the network switches, but don't see any problems.   Any suggestions?

dzenz · Answer

I would suspect a potential configuration conflict on the production network. I have some questions in order to better understand your environment.   - Is only out-of-band management being used? - Are the MD30000i management ports configured for DHCP or static IP's?   From this we can dig further.   Dave

PleaseUseEmailF · Answer

All iSCSI Network interfaces are on the 172.29.159.xxx/24 network. This network is not routed over our network at all and a dedicated switch is being used for these connections.

The production network is 10.96.159.xxx/24. The management interfaces live on this network.

I have a dedicated machine setup for iSCSI management using the out-of-band connections. I installed the management application on one of the iSCSI servers to test the in-band management, but was experience the same problems.

I have now found the problem gets worse when the MD3000i unit is under a heavy load. If I run the SQLIOSIM utility to test the unit, I am unable to access the management interfaces. If I kill the test and wait a few minutes, I can SOMETIMES get the unit back.

dzenz · Answer

Just to clarify, MDSM does not need to be installed on a iSCSI server that's connected to the array to utilize in-band. The management station with MDSM only needs to have UDP/TCP access to one (or more) iSCSI server and a proxy will manage communication to the array.    Additionally only out-of-band management could be used with MDSM installed on a iSCSI server as long as the host has UDP/TCP access to the management ports on the array.   Is Automatic or Manual discovery being used? Can you detail what the 'Out-of-band management connection(s)' and the 'In-band management connection(s)' on the Summary tab indicate? What OS's are running on the management station and host servers?   Dave

PleaseUseEmailF · Answer

I have tried both automatic and manual discovery modes. Both methods occasionally discover one of the controllers, but say the MD3000i can not be managed.

If I am able to make one of these “half” connections, the summary tab will show the IP address of one of the out-of-band management connections. But all of the tabs are blank and it says the array cannot be managed in this state. I don't have the specific error in front of me. (I rebooted the array earlier to regain access).

The management station and servers are all Windows 2003 R2 SP2.

Thanks for your help so far.

dzenz · Answer

Have you configured static IPs for the controller managment ports or are you using DHCP?   Dave

PleaseUseEmailF · Answer

I am using DHCP, but using reservations in DHCP so the array will always get the same IP Addresses.

When this problem occurs, I am still able to ping the controller IP addresses (management and iSCSI). I never lose that ability.

dzenz · Answer

During boot the controllers in the MD3000i will attempt to obtain management port IPs via DHCP, if a DHCP server is not available then the following default IP's will be used.

controller 0 - 192.168.128.101 Subnet Mask 255.255.255.0

controller 1 - 192.168.128.102 Subnet Mask 255.255.255.0

Can you perform the following steps and report the results?

1 - Configure a mini-network using the 192.168.128 subnet (e.g. assign 192.168.128.001 as the IP on a system)

1 - Unplug the network connections from each management port of the controllers

2 - Restart the array (the controllers will default to the static IPs)

3 - Once the status LED on the MD3000i is blue reconnect the management port connections

4 - From the system that is configured with the 192.168.128.001 IP ping the two controller IPs listed above

5 - Perform an Automatic discovery in MDSM (discovery should succeed and both controller IPs should be listed on the Summary tab)

Dave

PleaseUseEmailF · Answer

I will have to get back to you on this test setup.  I will do it, but it might take a few days.   BTW, I lost management access to the array again over the weekend.  This morning I am completely unable to access it.  Automatic and Manual discovery do not find anything.   My servers are still talking iSCSI to it, but I can't manage it in any way.

PleaseUseEmailF · Answer

It looks like Dell has released new firmware and drivers for the MD3000i yesterday.  I am downloading them now and will try them out as well.

dzenz · Answer

The latest release includes support for SATA physical disks in the MD3000 (and MD3000i) storage array.   Even though you're using SAS physical disks if you have the opportunity to update the stack to the latest release please do and reply with your results.   Dave

PleaseUseEmailF · Answer

Things are looking much better now.

I reconfigured the network and reinstalled the OS on all of my servers.

I think the issue was with how I was connecting to the iSCSI array. All of Dell's documents recommend using 2 networks for the iSCSI connections as described here:

http://www.delltechcenter.com/page/MPIO+to+PowerVault+MD3000i+with+Microsoft+iSCSI+Initiator?t=anon.

I initially configured my SAN this way, but moved to a single subnet for all iSCSI connections. I think this caused the issue and is why 2 networks are recommended/required. When I would send large amounts of traffic to the array, I would lose the management console connections.

Now that I have moved to the "recommended" 2 network setup I have not seen the issue.
I am going to enable jumbo frames on the switches and NIC soon and run additional performance tests.

My setup:
Dell MD3000i - 15 x 146G 15k SAS - One Big RAID 5 - 3 LUNs
4 x Dell 1950 Servers (2 x 2 node Windows 2003 Clusters)
Intel Pro/1000 PT Dual Port NICs for iSCSI
Cisco 3750 Switches

dzenz · Answer

Good to read your latest post.   Regards,   Dave Message Edited by dzenz on 12-18-2007 04:22 PM

RMAPPJR · Answer

My issue is the same plus I have two MD3000's and an MD1000 who's out-of-band management is inaccessible.  One MD3000 is in a totally separate server room on a different Cisco switch.  All MD Management NIC's are VLAN'd to network 192.168.61.XXX which is where our Dell IP KVM's reside.  These are the only devices on the specified VLAN that are experiencing trouble.  Could the 3000i bring down all of the other array interfaces just because of the single iSCSI VLAN configuration ?

dzenz · Answer

I'm trying to understand your topology.   Could you post a representation of the associated IP's and VLAN tags in use?   Dave

RMAPPJR · Answer

(All Static IP Assignments)

VLAN 61 = Our IP KVMs, Remote Access Controllers and PowerVault MD Management Controller Ports

VLAN 116 = Our non-routed iSCSI traffic

VLAN 6 = Our Server LAN

MD3000i:

iSCSI Controller 0, In 0 = IP 192.168.116.31, Sub 255.255.255.0

iSCSI Controller 0, In 1 = IP 192.168.116.35, Sub 255.255.255.0

Management port, cntrl. 0 = IP 192.168.61.65, Sub 255.255.255.0, GTWY 192.168.61.1

iSCSI Controller 0, In 0 = IP 192.168.116.34, Sub 255.255.255.0

iSCSI Controller 0, In 1 = IP 192.168.116.36, Sub 255.255.255.0

Management port, cntrl. 1 = IP 192.168.61.66, Sub 255.255.255.0, GTWY 192.168.61.1

FileServer5: IP 192.168.6.161

iSCSI port 1, 192.168.116.37

iSCSI port 2, 192.168.116.39

PowerVault

Was this post helpful?