UNSOLVED
incremental sync duration increases exponentially
I want to thank Mr. Ito and Mr. Ramjass for replying to my previous post. One thing I've noticed about Replistor is that the time it takes to complete an incremental sync increases exponentially with the number of files going up. For example, my 3 specifications:
1) 1,642,817 files (3 hours 21 minutes)
2) 7,529,326 files (22 hours 50 minutes)
3) 13,119,422 files (50 hours 13 minutes)
The performance settings is:
TCP/IP Buffer Size: 64
Full Sync Buffer Size: 128
Maximum Batch Size: 256
Target Connections: 8
Disable Communications Compression: box unchecked
Again, we are using version 6.2 SP2. The files are tiff images ranging from 60k - 160k. Any users out there with some thoughts or suggestions are greatly welcomed!
1) 1,642,817 files (3 hours 21 minutes)
2) 7,529,326 files (22 hours 50 minutes)
3) 13,119,422 files (50 hours 13 minutes)
The performance settings is:
TCP/IP Buffer Size: 64
Full Sync Buffer Size: 128
Maximum Batch Size: 256
Target Connections: 8
Disable Communications Compression: box unchecked
Again, we are using version 6.2 SP2. The files are tiff images ranging from 60k - 160k. Any users out there with some thoughts or suggestions are greatly welcomed!
There is a change in the way RS does Syncs starting with 6.4, SP1. There is a bias now toward Mirroring activities. This is targeted at minimizing the number of OC$ files created during the Sync process. It will start placing the Sync Copy commands at the end of the queue instead of as they are enumerated. This allows the Mirroring to continue during the sync. However, it may (probably will) result in the initial syncs taking longer to complete. Once the sync is complete mirroring continues as it has in past releases. If the rate of change (mirroring operations) is too high the sync process can take longer. There is no way to anticipate the increase in time that a sync may take but it should eventually complete and result in fewer OC$ files being created.
Replistor has a version that is qualified on Windows 2008, x64. Just make sure us download the installer for x64. However, you must use the most recent version. Windows was not fully supported until 6.4.0, The latest version is 6.4.1.4 and should be used. If that version is not on the Powerlink download site then it can be provided on the ftp site by contacting someone in support.
- Hi,gjwl
When there are many small files, the reading and writing becomes slow.
This is specifications of the NTFS.
Therefore, it will not be improved even if I do any kind of performance tuning in RepliStor.
Because the bottleneck is in the NTFS.
I make 100,000 files of 100kb in my test environment and synchronized, but a transfer speed is only 3---5MB/sec.
This becomes slow for specifications of the NTFS as having said earlier.
Even if this is any kind of replication tool, it will be the same result.
jschrade1
2 Intern
•
106 Posts
339
0
Posted January 4th, 2010 10:00
gjwl:
There have been a number of improvements in the replication performance of RepliStor (RS) since 6.2. If you are replicating mostly small .tiff image files and they do not change often then I would recommend you try using Incremental Syncs with Attribute Compare Only turned on. Without the Attribute Only turned on, RS will open each file and create CRC checksum values to make sure the files are the same. Attribute Compare Only will only compare the Last Accessed Time/Date and the File Size. That is done very quickly, compared to the CRC checksum type comparison.
You may also change the HKLM\System\CurrentControlSet\Services\RepliStorControl\Parameters/DirEntrySplitLimit to 1000. The default in earlier versions was 100000. This controls the buffer size that RS creates when enumerating the files that must be replicated. This is used for comparing to the Target and is used to Delete Orphans on the Target node. We have found there is also some performance effexts as well. For this to take effect, you need to stop the RepliStorServer service, make the change and then re-start the service.
If you are running Windows 2003 SP1 or newer, I would recommend you upgrade to either the latest 6.4 version or the latest 6.3 version and try the above "tweaks" as well.
Thanks...JS