Unsolved
This post is more than 5 years old
1 Rookie
•
28 Posts
0
1778
isi_vol_copy -full failing when maxino value is reached
I am trying to migrate data from a NetApp filer to an Isilon cluster running OneFS v7.1.1.7 and it keeps failing. Sample output follows:
isilon01-1# isi_vol_copy /ifs/ -sa root:xxxxxxx -full
Migration Checkpoint Window: Files=100000 MBs=1024 Mins=60
maxino = 33685505
Processing entry:./Collateral/purge/43131/10000024.tif ino:29079524 total processed:100000
Extracting file:./Collateral/purge/43299/20000939.tif ino:2947864 total extracted:10000
Extracting file:./Collateral/purge/43656/20000027.tif ino:6103473 total extracted:20000
Extracting file:./Collateral/purge/43312/20000073.tif ino:9098270 total extracted:30000
Extracting file:./NST01/20NSTARFM153200002/11000374.jpg ino:12001458 total extracted:40000
Extracting file:./ibml_slg/processed/JPM90_IRVSCN11_014583.slg ino:14762545 total extracted:50000
Extracting file:./Collateral/purge/43309/10000708.tif ino:17721438 total extracted:60000
Extracting file:./ibml_slg/processed/EMC90_IRVSCN04_000115.slg ino:20966145 total extracted:70000
Extracting file:./ibml_slg/processed/JPM90_IRVSCN03_015964.slg ino:23756330 total extracted:80000
Extracting file:./ibml_slg/processed/JPM90_IRVSCN15_007160.slg ino:26825056 total extracted:90000
Extracting file:./NST01/20NSTARFM153200002/11000670.jpg ino:29962577 total extracted:100000
Extracting file:./NST01/20NSTARFM153080001/10000863.tif ino:33133128 total extracted:110000
The transfer aborted with error.
isi_vol_copy did not run properly
The first time I ran isi_vol_copy, I attempted to copy the entire 931GB volume and it ran for 24 hours, copying 927GB before it failed. I tried restarting the failed run, but it failed again in the same place. I then attempted to copy a 160GB subdirectory of the same volume to an empty test directory on the Isilon. It also failed to complete with the following error:
Extracting file:./pgp/1st Project/Done/pdf10/NSM-0255409299-REMAINING/10030815.tif ino:33585167 total extracted:2900000
The transfer aborted with error.
isi_vol_copy did not run properly
I am wondering if this is failing because of the maxino= 33685505 limit being exceeded. Where is this limitation from ? The NetApp filer or the Isilon ?
Each time the transfer has failed, it was extracting inodes close to that limit just before failing.
crklosterman
450 Posts
1
December 14th, 2015 07:00
leemcfarland,
3 Things,
1. isi_vol_copy is a supported component of the product. As such, you are certainly welcome to submit an SR to support.emc.com and they can help you debug the issue.
2. isi_vol_copy is an NDMP dump, if you change the max number of inodes between a full and an incremental pass with the OnTAP engineering mode commands, I've seen this sort of failure before. If the volume is not huge, I would suggest simply deleting the target dataset, and then re-running a full copy. You can do this really quickly with 'isi job start treedelete --path=/ifs/path'. Be extremely careful & surgical with a treedelete; it is very fast and very effective, and entirely un-recoverable (except for snaps or offline backups).
3. If the volume in question has not changed between full and incremental, you may be encountering a limitation of isi_vol_copy without some tweaking. Specifically that it stores the logs in the boot mirror (in /var) of the node running isi_vol_copy. While that works 95% of the time, if you have a volume with more than about 40 million files, the logs could fill that boot mirror. Support has a process that can help you relocate those log files from the boot mirror off to /ifs/.
Read that KB Here: https://support.emc.com/kb/209370
Hope this helps,
~Chris Klosterman
Advisory Solution Architect
Emerging Technologies Enablement Team
chris.klosterman@emc.com
twitter: @croaking
Lee_exelatech
1 Rookie
1 Rookie
•
28 Posts
0
December 14th, 2015 19:00
Thanks for the reply Chris.
RE:1 This cluster is not on any support agreement yet. It is being re-purposed after being moth-balled for a few months. I may have to get it on an agreement so I can get this migration completed.
RE:2 The NetApp volume is under 1TB, so it is not huge. I tried re-copying the volume to another destination folder on the Isilon but it still failed. I increased the maxfiles setting for the NetApp volume and it did increase the maxino= above the point where the copy was failing, but the full copy still fails. I also tried a full copy of the individual subdirectories in the volume, but it still fails to complete.
RE:3 I have redirected the logging from /var/log to /ifs/data. I ran into the /var filling up bug on another migration last month, so I made that change at the start of this migration.
I may have to throw in the towel and get support involved.
scott_owens
60 Posts
0
December 15th, 2015 14:00
How many files does the source volume have? You can get that output by looking at the output of the following on the source netapp.
'df -i'
Lee_exelatech
1 Rookie
1 Rookie
•
28 Posts
0
December 15th, 2015 15:00
IrvingNetapp1*> df -i images
Filesystem iused ifree %iused Mounted on
/vol/images/ 3972873 41027126 9% /vol/images/
I was able to get a subfolder of a subfolder in the volume to complete a full isi_vol_copy (/vol/images/scanning/images/NST01), which was ~18k files, 5 folders and 1.6GB.
Phil.Lam
1 Rookie
1 Rookie
•
573 Posts
1
December 15th, 2015 21:00
Is this multi-protocol? If NFS, have you tried rsyncing the directory that has the error? If SMB, have you tried EMCopy or Robocopy?
Lee_exelatech
1 Rookie
1 Rookie
•
28 Posts
0
December 16th, 2015 16:00
It is SMB. I have not tried Robocopy or EMCopy yet. I just tried isi_vol_copy'ing the subdirectories over one at a time over the last couple of days and was successful on the smaller (<10GB, <1M files) subdirs. I just tried one that was about 15GB and 2M files but it failed in much the same manner as the full volume copy attempt did. Of all the failed attempts, none have failed in the same place.
I am abandoning the isi_vol_copy route and will have to try Robocopy. I have been doing these NetApp -> Isilon migrations using isi_vol_copy for the past year and this is the first one that has failed like this.
Maybe I'll have better luck with Robocopy.