How Centera replicates delete and other questions

Hi Naty

Before responding too quickly would you mind to provide a show pool capacity output as well?

Your interpretation is correct. This is what the queue contains. I have had instances where the queue was somehow containing from each access node the clips to be replicated and thus reported roughly double of what was really to be replicated. Order to understand if you really have already written that much the pool capacity output would help.

Are deletes included in this total amount of MB and how?

Replication is a time ordered queue. I not 100% sure what happens when you have an object that is in the replication queue and then delete it.

Is there a way od knowing how much MB comes from each pool?

Not exactly. You could based on the amount of outstanding objects per pool and their average object size (calculate from show pool capacity) estimate how much data is pending for which pool.

What would happen if we stopped replication of one pool, in order to temporarily releave some BW for other pools?

Would data that is already in the queue for replication have to be transfered anyway?

When you remove a pool from replication it's object still in the queue will be removed. You would have to perform a restore when you start replicating the pool later in order to ensure all data is really on the target system.

I am trying to find a way to replicate at least data from some (more important) pools, and give us some time to negotiate more BW or something else.

Remove the other pools from replication see above.

Or disable replication at all. Then re-enable it for the most important pool (or all pools depening on new data write rate) and perfrom a pool per pool restore for the already written data.

IF you want me to quickly have a look provide a Teamviewer or NetViewer session details. Then we can have a chat at what steps would help you best.

Best regards, Holger

The amount of data to be replicated is an approximation of the total size of clips and blobs in the replication queue.

This does not mean that this is the amount of data that will be replicated as some of these objects may already exist on the replica nor does it compensate for single instancing that may occur.

Objects touched by regeneration or organic internal Centera processes can cause these to be requeued for replication when they have already been replicated. If the 'exists' right is granted to the replication profile on the replica, these objects will not be replicated (well, they may depending on the size).

Deletes are not calculated in the size.

There is no way of knowing the size of the data to be replicated from individual pools.

You cannot disable replication of some pools and not others without losing the objects that are queued for replication now or will be queued while disabled. This will cause the clusters to become out of sync and I do not recommend doing this.

If you wish in speed up replication, EMC Centera Support can increase the amount of clip exporters (threads) used by replication. This in combination with adding additional replication role nodes should kick it up a notch.

Yes, I wouldn't like new problems as well. But increasing nuber of threads or adding additional replication role nodes will not help if the link is full, right? The way I see it now we should just stop writing and wait for better link :-)

Thank you Jacob, Dennis.

Unfortunately, I don't have either Teamviewer nor NetViewer .

Regarding how much MB comes from each pool, I already tried the way Jacob suggested, but when I calculate the sum, it doesn't match 'MB to be replicated'.

Here is display of 'show pool capacity':

Capacity / Pool      Quota          Used           Free        C-Clips     Files
------------------------------------------------------------------------------
KLDpool                 6,144 GB   4,048 GB   2,096 GB   5466895     5466895
Arhiva                    6,144 GB   3,857 GB   2,287 GB   7551        7552
MASpool                5,000 GB   3,563 GB   1,437 GB   4884        731936
TSMpool                7,000 GB   4,634 GB   2,366 GB   1274354     1274357
MASVZpool          18,000 GB  16,750 GB  1,250 GB   70009       3490333
AuditArchive              --          85 MB              --         63696       0
SystemArchive             --         0 GB               --         0           0
default                        --         487 GB             --         1557        86366
ConsoleArchive            --         0 GB               --         0           0
                                                                             
Total                          --         33,340 GB         --         6888946     11057439
------------------------------------------------------------------------------

I am thinking about what Dennis wrote - if I disable replication for a pool, what would happen with data that is deleted in the mean time - only on local Centera, but not on remote? It would only work for applications that only have writes - then I could easily restore new writes afterwards. Or not?

If you disable replication on a pool, everything that is in the replication queue from that pool will be removed from the queue and not replicated. Any new writes or deletes made to that pool will not be replicated.  You use "restore" to restore any data written within a specified time period but I do not believe that restore will copy a delete but I could be wrong

I would rather see you take an approach that fixes the problem rather than works around it and potentially creates other problems.

One other additional comment on your original post - The size of what needs to be replicated can become very skewed if there are any incomplete databases such that a DBinit was running

Replication will consume as much bandwidth that is available for what-ever action it is performing regardless of the thread count. The thread count (clip exporters) can be set to 1 and you are still maximizing network bandwidth.

You could try increasing the clip exporters and then monitor to see if things progress faster however I would not recommend blindly increasing the clip exporters as this increases the connection count which as a general rule should not exceed 20. If the replication nodes are also access nodes, this could push you beyond this accepted threshold. If the replication nodes are replication-only nodes then you should be able to safely increase the clip exporters.

This would require EMC involvement