I have actually had a similar argument with a colleague. I didnt really have a theory on how they communicated, I just thought it would be a poor design to rely on an outside network to do SP to SP management communication (why not just through a serial bus between the SPs).

We couldn't find anything to show the actual design but since we are migrating off the array in September we thought that once everything is disconnected we would remove the network connections and serial into 1 SP. And if we could still see/manage the other SP we would feel comfortable that there was something there to facilitate the communications.

Brian,
This should settle in on the Cx arrays at least.

From the white paper titled 'Introduction to Cx3 UltraScale Series' available on PowerLink:

"There is a PCI Express-x8 CMI channel between the storage processors that is use for comunication and messaging between the storage processors as well for the mirroring of data that is written to the portion of memory that has been designated for write cache on the storage system."

The management ports are 100Mb ethernet. How does your boss think two high speed storage processors are going to mirror 8GB of cache and process array communications over a 100Mb ethernet link in addition to all the external mangement interactions?


Aran

Actually I have to correct my statement. Since it is only write cache that is mirror, only about 3GB of cache is being mirrored between the SPs on the biggest Cx3, the Cx3-80.

Message was edited by:
AranH

The only documentation I found to support this (with a picture) was from EMC training material that I'm not allowed to copy, but I can assure you that on the CX3 (and CX & FC for that matter) the two SPs communicate through an internal connection completely independant of the external network connectivity.

I'm not sure about the AX as I've never worked on them, but the higher end arrays don't rely on outside network connectivity for management. This only applies (obviously) within a single array though. If you have multiple arrays in a domain, they talk to each other over IP.

There are two methods that the SP's use to communicate (this is for the CX series as I understand it):

1. internal bus (CMI bus) - this is used to send the mirrored write cache data between the two SP's and also other checks - when one SP fails, the other SP detects this over the CMI bus. It's also used for other internal communications

2. Management Ports - these two Ethernet ports are used for management functions - the Management Server (the process running on each SP) communicate using this link - when the management ports can not talk to each other, you can not "manage" the SP's - bind LUNs, etc. These functions (management) use the management ports. If the SP's can't talk to each other over this link, then you will see in Navisphere UI, the notice that the other SP is "unmanaged". So yes, you must have these two ports talking to each other.

To be fair, I have not tested this on the CX3-series and they may have changed something, but it's this way for the CX200-700.

glen

Glen,
Thanks for jumping in, I was hoping one of you guys would chime in on this. I understand the caveat listed in bullet 2, but that is for management functions (monitoring, storage allocation, etc... ) not internal array operations. So for the sake of argument the critical array functions would not be affected by the Management Ports being disconnected from the LAN right?

Aran

Mr. Kelleg,

You are correct. With an AX array, if the storage processors cannot see each other across the external LAN they will generate an error reporting each other as being unmanaged. On my CX array, if you unlug one of the Ethernet ports you don't see any errors.

I agree with those people who have responded to this thread and said that the "high-end arrays" communicate internally and don't care if you lose your network connection.

My supervisor says that losing network connectivity to either SP on a CX3 should generate an error message. He tells me he submitted a bug report to Dell to see if this is a Flare problem.

Let's think about this for a moment... If the CX3 is designed tell you you lost your management connection, and it doesn't, that's a problem. If the CX3 is designed not to alert you that you lost your management connection, isn't that also a problem? In my case the system was disconnected for 2 days and I had no idea anything was wrong.

Maybe I could fit my boss with a collar that has an RFID chip, so I get an alert anytime he gets too close to my SAN. But maybe a simpler solution would be for the CX to alert you when you lose your network connectivity.

Many thanks to all you brilliant folks who took the time to respond.

(BTW, things here in Baghdad are going well.)

Brian

Thanks for serving Brian, I hope you get to come home soon.

Aran

Are fault notifications critical functions?

Oh, come on Aran... we both know that nothing is "critical" to the business unless they lose access to their data. We could have an array with no LAN connectivity one failed SP and half a dozen hot spares running (but 10 drives failed) and the business would be happy!!!

What they don't know won't hurt you, right?