how to add more than one nodes to Cluster

Hello Ayas,

You can add one node at a time  once node is added Multiscan job will start automatically which has 2 sub jobs (collect and autobalance) so motive of running  autobalance will be taken care automatically.

Hi chughh

Thanks for the quick update !

So we do not need to wait for Multiscan to complete before adding another node right ?

Thanks

Aya

Hi Rob …

Thanks for the clarification !

I was confused with some relocation process slow down

Volume access … but as you say , it is not a worry when you think

About initial set up ..

Thanks you !

Aya

As others have said, you don't have to wait.  However, I've seen nothing but badness adding only 2 nodes to the cluster, especially if the rest of the nodes in the cluster are close to full.

AutoBalance takes *forever*.  You can add a node in a minute as the marketing folks say, but it can takes many days for AutoBalance to complete.  You may only move a few TB per day which isn't much if you're adding 144TB nodes.  I added 4 x 144TB nodes to a fairly full cluster last Thursday.  Each of the node nodes has only 8.5TB on it yet the original nodes have >115TB and that's with 5+ days of AutoBalance running, most of that with a high impact policy.

Additionally,if you add 1 or 2 nodes, the data can't be protected without writing to the remaining nodes so if you're close to full, you'll may get yourself into trouble before the space is balanced.  I've run a cluster out of space doing this - the data was coming in faster than autobalance could move the data - and it took IntegrityScan over 6 months to finish as a result.  Add 3 or more nodes if you're using 2:1 protection so you can use all of the new space immediately.

as a best practice you should wait 60 seconds between node adds. That is the only time that should be needed to add additional nodes to the cluster. I would not reccomend trying to add two nodes at the exact same time as I have seen issues with that, ghost nodes in a cluster, Mergelocks though not very frequent those can happen. The best mitigation is to give 60 seconds between node adds that is all the time that should be needed to succesfully add more than one node to a cluster but never more than one at a time.

My normal procedure is to write down the last 4 digits of the serial numbers of the nodes I'm going to add so I know which order to add them in.  Then I add 1, wait until it is downloading any software it needs and reboots, watch it go ready, then go onto the next.  The process can probably be sped up, but if anything goes wrong, I don't want too many things in flight, and I don't do this every day anyway.  So what if it takes me an hour or so to add 4 nodes?  I've added a lot of nodes in the last couple of years and am up to 5+ PB of Isilon storage now.  The process hasn't failed me yet.

Ed, which OneFS version is this?

We didn't have too bad experience when adding one NL400 node

to a 80% filled 7.0.2.9 cluster recently.

That is, MultiScan got system cancelled on a group change "as usual",

so we let SmartPools finish the job and ran Collect on LOW impact later.

MultiScan and then SmartPools took three weeks altogether, but

on LOW/MEDIUM/PAUSED impacts, likely LOW on average.

Based on the 6.5.5 experience I had expected five to six weeks.

Cheers

-- Peter

Hi Ed,D_tracy ,Rob and chughh

Thanks heaps for the great info! it is so good to know what everyone has been through .

Actual fact is strongest info !

thanks All ! it is great !

Aya

I had the issue the first time in 6.5.5.something.   It's still an issue today with 7.1.0.6 - it's extremely slow.80% usually isn't much of a problem if your ingest rate isn't too high.  It's when it gets into the 90s that I really start to worry.

Rob's suggestion of using pools isn't a bad one but I've seen SmartPools take forever to run too.  I've got a job that's been runing for nearly 3 days on what I consider a small cluster (14 nodes, 800TB total, with 233TB used).

I do need to figure out how to retrofit existing NL nodes with SSD drives once 7.2 is actually production ready a year or so from now.

Thanks and good luck then! Peter