MD3200i and powerconnect 5524 with ISCSI and TCP

I see and perform MD setups fairly frequently.

Isolate the 2 switches from eachother (don't link them together (stacking or otherwise)) and dedicate them to iSCSI.

Then you need to use 4 subnets for iSCSI. You can stick with the factory defaults (192.168.130.x to 192.168.133.x), but you can easily also change these if they conflict with something in your LAN.

Each switch will handle 2 of the subnets.

If your servers have 8 or more network ports, you could have each server use 2 connections to each switch. If your servers have fewer than 8 network ports, you just use 1 port to each switch and spread out the hosts between the 4 subnets.

Big note: servers can only share SAN storage if they are using some kind of clustering solution (e.g. Microsoft Cluster Services), or a cluster-aware filesystem (e.g. VMware's VMFS or RedHat's GFS).

To give you an idea how this would go subnet-to-switch wise:

Switch 1 will handle subnets 192.168.130.x and 192.168.132.x:

Connected:

Controller 0 iSCSI port 0 (192.168.130.101)

Controller 1 iSCSI port 0 (192.168.130.102)

Controller 0 iSCSI port 2 (192.168.132.101)

Controller 1 iSCSI port 2 (192.168.132.102)

Server 1 iSCSI NIC 1: 192.168.130.200

Server 2 iSCSI NIC 1: 192.168.132.201

Server 3 iSCSI NIC 1: 192.168.130.202

Server 4 iSCSI NIC 1: 192.168.132.203

Server 5 iSCSI NIC 1: 192.168.130.204

Switch 2 will handle subnets 192.168.131.x and 192.168.133.x

Connected:

Controller 0 iSCSI port 1 (192.168.131.101)

Controller 1 iSCSI port 1 (192.168.131.102)

Controller 0 iSCSI port 3 (192.168.133.101)

Controller 1 iSCSI port 3 (192.168.133.102)

Server 1 iSCSI NIC 2: 192.168.131.200

Server 2 iSCSI NIC 2: 192.168.133.201

Server 3 iSCSI NIC 2: 192.168.131.202

Server 4 iSCSI NIC 2: 192.168.133.203

Server 5 iSCSI NIC 2: 192.168.131.204

If you're running Windows, you'll need to manually define each of the 4 available connections for the 5 servers. If you're running VMware it will automatically connect to all the available ports. I think linux also automatically connects to all available ports (and you may even have to explicitly remove the unreachable ports to keep it from trying to connect to a port it cannot reach due to not having a NIC with an IP in that subnet.

hi,

thanks a lot:

so i'm not far from your organization: I took default ip adressing from the MD storage, and use subnet for NIC configration except that: switch 1 manage all subnet assigned to controler 0 and switch 2 to controler 1.

Is this really bad?

switch 1:

Controler 0 192.168.130.101

Controler 0 192.168.131.101

Controler 0 192.168.132.101

Controler 0 192.168.133.101

Switch 2:

Controler 1 192.168.130.102

Controler 1 192.168.131.102

Controler 1 192.168.132.102

Controler 1 192.168.133.102

Server 1:

192.168.130.1  to switch 1 controler 0:  192.168.130.101

192.168.130.2  to  switch 2 controler 1:  192.168.130.102

Server 2:

192.168.131.1  to switch 1 controler 0:  192.168.131.101

192.168.131.2  to  switch 2 controler 1:  192.168.131.102

etc ...

the problem in my configuration is that can't dedicate switch to ISCSI trafic "only"..

I have data also (netwrok, web, etc..)

I plan to use "keepalived" cluster to manage failover.

I do not plan any loadbalancing, loadsharing on the disk array,

only failover switching mode between the server and a backup server that act as a spare server.

also my 2 switches are stacked..with hdmi cable..shall I remove it?

If I do so, I shall assign another ip to the second Switch ?

how then shall I configure the NIC (2 per serveur) to access network, I cannot bond the NIC?

how do I configure gateways? nic 1 to switch 1 and nic 2 to switch 2?

how do I configure the balance between the 2 switches?

and how do I configure my application? which IP shall I use?

maybe I miss somehting here in my network knowledge.

thanks in advnace for your lightnings ;)

best regards

vincent

You're going to want to change that cabling.

A single subnet has to have all ports to a single switch, so you cannot have 192.168.130.101 (controller 0 iSCSI port 0) and 192.168.130.102 (controller 1 iSCSI port 0) to 2 different switches. This is an absolute requirements.

Also, in your setup, if you lose 1 switch you lose total access to one of the controllers. So if you somehow don't notice the faulty switch and then have a raid controller failure or you restart that controller for a firmware upgrade, you'll lose total storage access.

My example would allow you to lose 1 switch as well as 1 controller and still be online (reduced bandwidth, but still operational). This combo should be rare, but why not do the setup correct from the start.

You don't set a gateway on the iSCSI network ports on the server(s). They only need a gateway on the LAN side. If you purchased the replication license for the SAN (as well as a 2nd MD SAN with a replication license), then the SAN may need to have a gateway defined on one or more of the iSCSI ports, but again; not on the host (iSCSI) ports.

Your 5500-series switch is not a very powerful switch. So by running LAN as well as iSCSI on it, could impact iSCSI performance. I would only start considering multipurposing iSCSI and LAN if you had at least a PowerConnect 7000-series (or Force10 S55 (keeping within the Dell options here as this is a Dell message board)).

If you need the stack for the LAN side, I'd suggest to use 4 VLANs for iSCSI; one for each subnet, and don't let them cross the switches.

As an example:

VLAN 130: Ports gi1/0/1 to gi1/0/4 (2 ports from the array and 1 port from each of 2 servers) - untag the ports (switchport access vlan 130)

VLAN 131: Ports gi1/0/5 to gi1/0/8 (2 ports from the array and 1 port from each of 2 servers) - untag the ports (switchport access vlan 131)

VLAN 132: Ports gi2/0/1 to gi2/0/4 (2 ports from the array and 1 port from each of 2 servers) - untag the ports (switchport access vlan 132)

VLAN 133: Ports gi2/0/5 to gi2/0/8 (2 ports from the array and 1 port from each of 2 servers) - untag the ports (switchport access vlan 133)

Do not assign any IPs to these VLANs unless you plan to use MD-based replication.

Hi,

i follow your recommandation.

remove the staking hdmi cables,

assigned 2 ips to switches: 192.168.1.19 and .1.20.

Controler 0 192.168.130.101 switch 1  port 2

Controler 1 192.168.130.102 switch 1 port 4

Controler 0 192.168.132.101 switch 1 port 6

Controler 1 192.168.132.102 switch 1 port  8

Controler 0 192.168.131.101 switch 2 port 2

Controler 1 192.168.131.102 switch 2 port 4

Controler 0 192.168.133.101 switch 2 port 6

Controler 1 192.168.133.102 switch 2 port 8

and changed server ip like that:

APP1

192.168.130.1  to switch 1  port 14

192.168.132.1  to  switch 2 port 14

APP2

192.168.131.2  to switch 1   port 16

192.168.133.2  to  switch 2  port 16

DB1

192.168.130.3 to switch 1  port 18

192.168.132.3 to switch 2  port 18

DB2

192.168.131.4  to switch 1   port 20

192.168.133.4  to  switch 2  port 20

server 5 (not connected yet..future spare backup)

192.168.130.5 to switch 1   port 12

192.168.132.5  to  switch 2   port 12

is this correct?

unfortunatly, the hardware configuration is freezed, and I have to do with it:

2x 5524 switches.

now on each device,

if I run iscsiadm -m session,

I see 4 sessions:

[root@moobdb1 iscsi]$ iscsiadm -m session

tcp: [1] 192.168.130.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

tcp: [2] 192.168.130.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

tcp: [3] 192.168.132.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

tcp: [4] 192.168.132.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

[root@moobdb2 iscsi]$ iscsiadm -m session

tcp: [1] 192.168.131.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

tcp: [2] 192.168.133.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

tcp: [3] 192.168.131.101:3260,1 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

tcp: [4] 192.168.133.102:3260,2 iqn.1984-05.com.dell:powervault.md3200i.6f01faf000e2b00c0000000052c6830d

how should I configure VLAN from this point ? I read that I should tagg the port, and you propose not..i'm a bit lost..

I have LAN connected also: 6 server (2 web front, 2 application, 2 DB)

I have bonded the 2 first NICS of my server and will connect each on each switch.

routeur gateway is 192.168.1.1

switches: 192.168.1.19 and 192.168.1.20 on port 1 both.

I plan to use the web server as failover server with kginx + keepalied, or LVS+keepalive. maybe partition sharing, but not sure it's necessary. Only web server will have a public IP access, and forward to web applcation server.

all server will need to be accessed  for maitenance purpose in ssh.

but I think I have first to solve first iscsi config.

thanks for your support.

vincent