Testing new node types in existing cluster

Hello Hershal,

for the most part there is no real testing needed, in best case scenarios you are able to plug the node into power (give it at least an hour to charge nvram batteries) then attach network cables and join to cluster.

I am aware of some customers that allow a "burn in" period where they stand new nodes up in a cluster of 1. meaning that they run through the wizard and set up a dummy cluster with no network connectivity. This allows for all hardware to be powered on and disks to spin up. The customers I am aware of doing this generally run it for three days in this configuration.

advantage:

This allows any hardware failures that may be a result of DOA, heavy handling on shipment, etc. etc. to burn out or fail before adding it to a production cluster and potentially having that impact production.

pitfall:

takes more time to add capacity, RAM, and CPU to cluster. (this method assumes we are not in a rush to add capacity or CPU cycles to existing cluster immediately)

We tend to add a new node(s) yearly, for the most part - we just plug in and go-live.

They pretty much *just* work without any issues, *jinx myself before this years upg*.

Good luck though!

_LEO_

Have you ever had any issues with the nodes when you plug in and go live?

Also, is there a reason you only add them yearly?

I am getting ready to do the same, so any information helps.

Thanks

-MW

Never any problems, just plug and go.

Oh, once we had to quick quirk where one new node needed to be properly formatted rather then the system just automatically being added in.

We only get funding per each year to add in.

So far we've added more NL400 108s then a bank of NL400 144s, this year hopefully a bank of HD400's to the cluster.

They just work....its great!

If you currently have no metadata acceleration to SSDs, the HD400s should be just fine for you. If you do, be aware that it doesn't work and will never work with the HD400s and NL410s, as those nodes SSDs are mandatory L3-metadata only and GNA will not be used with them.

Ahh - ok, I forgot that we've also got 3x S200's for acceleration too.

Yeah, we found out the really hard way when people started complaining at all levels about everything being ridiculously slow, including tab completes and logins. metadata-only L3 is evil. Needless to say, the NL410s are not in that cluster any more.

We have HD400s in our backup target cluster, and they seem just fine there, albeit pretty darn slow.

We have been told by Isilon that the metadata-only L3 SSDs are there pretty much exclusively to make background jobs not suck so badly.

carlilek,

why were the A100s terrible? we may want to spice up our custers with some of them...

we usually also add the nodes, but do not give them IP configs until disk/node firmwares are updated. If everything that might Impact the user is finished the ports get an IP and are available for the user.

We mostly rely on SMB and have no transparent failover (SMB3) yet. If we achieve this we might be able to just plug and go

Hi sluetze,

I think we've already established that you and I have rather different workloads, so it may be that the A100s would work in your environment. That said, word from EMC is that the A100s will be discontinued fairly soon and will not be replaced.

Our issue with the A100s was twofold; they are not a well thought out product, and they negatively affect clusters with nodes with smaller memory size (ie, all of them...)

1. For "accelerator" nodes, their CPUs are pathetic (much like all other Isilon CPUs). EMC does a good job of trying to save a buck by using the crappiest Xeons they can get their hands on, despite the fact that Isilon is toooootally CPU bound. The A100s have a pair of 6 core 2Ghz Sandy Bridge Xeons. So they don't exactly add a lot of CPU to the cluster.

2. They are exclusively L1 cache. So if ALL IO is going through them (and your IO doesn't exceed the flush rate of ~240 GB/node), they would be awesome. But by definition, they can only accelerate files that they already have in their cache--they don't do anything at all for connections to other nodes. Additionally, that means they rely on caching making a difference in the workload; if your workload is all random with very few re-reads of data, they're not going to do a darn thing. Other than....

3. 256GB is more memory than you can get in any other kind of node (aside from S210s and X410s). OneFS has an ongoing issue with group changes where if the memory sizes are not equal, locks are not distributed properly and the cluster essentially completely stalls for a period of time. This has improved in recent versions, but it's not entirely gone. At the time we were using the A100s, our cluster had nodes in it with as little as 12GB of RAM, and we would see that group change stall take up to 3 minutes. And when I say locks, I mean the cluster doesn't serve NFS, doesn't accept new mounts, doesn't serve SMB, doesn't update the WebUI, and even on the command line is completely inaccessible. In 7.2.1.1 with a delta of 48GB-128GB nodes, we still see stalls of something like 10-30s on group changes.

HI carlilek,

thanks for the information.

we currently are facing some problems with session counts and the CPU. the I/Os are not a problem and the spindles "chill" (thx to L3 Cache). Our thought was too increase CPU power without the necessity of creating new node-pools. Also we have enough capacity left in the cluster.

so our usecase would be to:

make A100 the "leading" node for internal jobs

get a lot of the sessions to the A100 to free cpu cycles on the other nodes (x200).

what exactly do you mean with "group changes"?

It's not actually related to drive stalls, it's the system shuffling all the internal locks and other items in RAM from the departed node to all other nodes in the cluster, or conversely rebalancing locks to the newly added node.

We have seen the drive stall issue, but that was quite some time ago (prob on 6.5.5.something).

I'm going to have this wrong since its been awhile for us, but have you tweaked the stall value for drives? We were having an issue where drive stalls were triggering group changes and support had us increase some stall value.  I believe its been adjusted in recent versions but can say for sure. I can't say we saw the same locks you did, our clusters are not that large though.

not enough.

24GB

Group changes are the background processes that Isilon goes through whenever a node or drive is added or removed. Fortunately, this issue only hits with node level operations (add, remove, reboot, shutdown). And of course, with a reboot, it's a double whammy since it shuts down (ie, is removed), and then comes back on (ie, is added).

How much RAM do your X200s have?