This will be one time task, once completed each cluster will have its own IB switch peers.

Per this 2nd cluster will be the new cluster (with brand new nodes), alternatively we can add new nodes to the present cluster then set these "particular folders" to rest on new node pool (yes, smartpools needed) and once smartpool jobs completed we can split these nodes and the data from the cluster with a "appropriate way".

BUT what could be the "appropriate way"?

Mr Zorba,

  In order to utilize the IB network for the transfer of data, you would have to join the nodes together into a single cluster, move all data to the new nodes, then smartfail the old nodes from that cluster.  This would work if the 500TB of which you speak represents ALL data, as the smartfail operation would remove data from the old nodes.  You could then use the old nodes to setup a 'new' empty cluster to use as you wish.

To be sure that I am explaining this correctly, I will summarize again in an example:

• 5 108NL nodes holding 500TB of data.
• 5 new NL400 nodes (720TB capacity)
• Add NL400 nodes to existing cluster - bringing cluster to 10 nodes total.
• Smartfail 5 108NL nodes, moving all existing data onto the new NL400 nodes.
• 5 108NL nodes now empty and ready for configuration as a new cluster.
• 500TB of data now on new NL400 hardware.

Otherwise, using the IB network for the transfer of data between two clusters is not recommended or officially supported. There can be problems with the integrity of the data if two clusters are placed on the same IB switch.  Your best bet for copying data, would be using SyncIQ which will take advantage of all external network bandwidth you allow to move data with multiple parallel workers.  You can also monitor and adjust the performance impact if clusters are handling production workflows at the same time.

I would use SyncIQ over joining nodes to existing cluster and Smartfail.  Smartfail will take ages, speaking from personal experience on 108NL.