UNSOLVED
Looking for tips on trouble shooting Isilon Replication Policies
I'm new to the Isilon game, so forgive me if I seem a bit green....
I have several policies that are failing with this error
SyncIQ policy failed, A work item has been restarted too many times. This is usually caused by a network failure or a persistent worker crash>
It only happens on certain policies, some consistently, others off and on.
I do not think it's network related, but am wondering what would cause persistent worker crashes?
Can anyone give some advice on how to trouble shoot this issue?
Thank you
I don't think it's publicly visible. the 'internal' Bug ID is 132657. I was receiving those same errors, upon investigation we had over 280,000 snapshots on one cluster, and nearly 200,000 on another. All from a sync policy, all pending deletion(but not deleted). Needless to say it's many, many TB and it's not proving easy to clean up. I only hope this does not affect other enterprises the way it's affected us.
Katie,
thank you so much for pointing out some trouble shooting steps.
1. Pinging from the Host to the target cluster worked.
2. # Traceroute also worked fine.
3. I VERY much appreciated the port numbers. Telnet worked on all the ports. 5666 5667 2097 2098 3147 3148.
Here are two examples of Policies
This one works fine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Policy Name
Q-Daily-S01
Description
Q-Daily-S01
Enabled
Yes, policy is enabled
Action
Synchronize
Run Job
Every 1 days at 05:45
Last Successful Run
05:45 AM
Last Started
2015-07-28 13:53:29-0500
Source Cluster
Source Root Directory
/ifs/sxxxxxxx/xxx/SxxSxxx/Daily/S01
Included Directories
No value
Excluded Directories
No value
File Matching Criteria
No value
Restrict Source Nodes
Yes, run the policy only on nodes in subnet 'Replication', pool 'Replication'
Target Cluster
Target Host
xxx003.xxx.com
Target Directory
/ifs/sxxxxxxx/xxx/SxxSxxx/Daily/S01
Restrict Target Nodes
No, connect to any nodes in the target cluster
Target Snapshots
Capture Snapshots
No, do not capture snapshots on the target cluster
Snapshot Alias Name
SIQ_%{SrcCluster}_%{PolicyName}
Snapshot Naming Pattern
SIQ_%{SrcCluster}_%{PolicyName}_%Y-%m-%d_%H-%M
Snapshot Expiration
Yes, snapshots expire after 1 day
Advanced Settings
Worker Threads Per Node
8
Log Level
Notice
Validate File Integrity
Yes, validate file integrity
Keep Reports For
1 year
Log Deletions on Synchronization
No, do not log deletions on synchronization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~!
Here is one that fails
~~~~~~~~~~~~~~~~~~~~~~~
▪Policy Name
Q-Daily-Cxxx
Description
Q-Daily-Cxxx
Enabled
Yes, policy is enabled
Action
Synchronize
Run Job
Every 1 days at 05:15 AM
Last Successful Run
This policy has never ran successfully
Last Started
2015-07-29 14:52:13-0500
Source Cluster
▪Source Root Directory
/ifs/sxxxxxxx/xxx/SxxSxxx/DailyCxxx
Included Directories
No value
Excluded Directories
No value
File Matching Criteria
No value
Restrict Source Nodes
Yes, run the policy only on nodes in subnet 'Replication', pool 'Replication'
Target Cluster
▪Target Host
xxx003.xxx.com
▪Target Directory
/ifs/sxxxxxxx/xxx/SxxSxxx/DailyCxxx
Restrict Target Nodes
No, connect to any nodes in the target cluster
Target Snapshots
Capture Snapshots
No, do not capture snapshots on the target cluster
Snapshot Alias Name
SIQ_%{SrcCluster}_%{PolicyName}
Snapshot Naming Pattern
SIQ_%{SrcCluster}_%{PolicyName}_%Y-%m-%d_%H-%M
Snapshot Expiration
Yes, snapshots expire after 1 day
Advanced Settings
Worker Threads Per Node
8
Log Level
Notice
Validate File Integrity
Yes, validate file integrity
Keep Reports For
1 year
Log Deletions on Synchronization
No, do not log deletions on synchronization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
there is an oddity on my target Cluster - there are two identical folders:
/ifs/sxxxxxxx/xxx/SxxSxxx/Daily/ /(list of all current daily's )
/ifs/sxxxxxxx/xxx/SxxSxxx/Daily/ /sxxxxxxx/xxx/SxxSxxx/Daily/(list of daily's - though not all - less than 1/3 )
the second one has changes dated as of 2014-03-10 - everything has this as the last modified date - and it does not have the same items as the one above it. I find it odd that there are two identical names - and wonder if this is causing some confusion on replications part.
I can send more policies that are having issues - but essentially they are all alike - except for the path past /ifs/sxxxxxxx/xxx/SxxSxxx/Daily/
Tom
Hello TAGILE,
At this point, the complex nature of your issue makes me think a service request with Support would be your best option. This issue will more than likely require in depth log analysis and possibly packet captures while the policies are running. Please provide Support with all of the troubleshooting information that I had you go through in the beginning of this post.
To create a service request, you have a couple options:
1. Log in to your online account on support.emc.com and go to this page: https://support.emc.com/servicecenter/createSR
2. Call in to EMC Isilon Support at 1-800-782-4362 (For a complete local country dial list, please see this document: http://www.emc.com/collateral/contact-us/h4165-csc-phonelist-ho.pdf)
Hello. Did you get to the bottom of this?
Reason I ask - I'm having the same thing - I've had a few polices 'fail' on me - and once they do, they seem not to recover without a resync.
But whilst it's happening - I'm still having all the _other_ jobs running just fine.
I've found:
KB477493 which suggests it's a networking issue - but I'm pretty sure I can rule that out. (Hard to say exhaustively mind, but I certainly don't have a WAN accelerator or load balancer in the mix.... and isn't it supposed to be TCP anyway?)
But I have 6 policies, of which one fails _every time_ now, and the other 5 work fine.
Having had this happen before - I'm pretty sure if I reset the sync state, and rerun - it'll be ok again, but this time it's ~100TB of data, so I'd really rather not do that. At least, not every time this crops up, which is now seeming about once every week or so.In case it's of use to people having this problem in future - I found setting a 'files-per-second' throttle (on my OneFS 8.0.0.1) helped for some reason.
I don't know why this should be - but 1000 files per second throttle, alleviated the problem. (I don't know if it made it go away entirely, but it certainly made a job that _always failed_ work again.
rash_vyas1
17 Posts
1320
0
Posted August 29th, 2017 08:00
I ran into similar issue and I made following changes to make it work. Most important change was to use consecutive IP range (last step) on the target cluster.
1. Use Static pool on source and target cluster
2. Use SmartConnect Zone name on the target cluster and use that name in the SyncIQ policy on the source cluster
a. Verify that source cluster can resolve and ping SmartConnect zone name
b. Verify SyncIQ ports are open between source and target cluster
3. On the syncIQ policy use - Force Interface: Yes
4. Force Interface and net.inet.ip.choose_ifa_by_ipsrc goes together, so check with Isilon support to make this sysctl change to 1.
5. On the synciq policy use - Restrict Target Network: Yes
6.. Use consecutive IPAddress on the Target cluster -- We had above steps done and policy was still failing with same error in this thread. Making the last change, use of consecutive IPAddress on the target cluster fixed the issue.
Example -- I had 30 nodes with ipranges 10.10.10.1 to 10, 10.10.10.30 to 40, and 10.10.10.60 to 70. I removed 20 nodes from the SyncIQ static pool on the target cluster and used just one consecutive ip range 10.10.10.1 to 10.
Each problem is unique solution may or may not work. Always engage and consult EMC Support before making changes.
Thanks
Rash Vyas
Technical Account Manager
Dell EMC
johnsonka
130 Posts
6703
1
Posted July 24th, 2015 14:00
Hello tagile,
Thank you for your question! Can you let us know the results from the following:
1. Confirm that the cluster can ping the SyncIQ policy's target cluster.
2. Confirm that a traceroute can be completed to the target cluster.
3. Test to confirm TCP ports 5666, 5667, 2097, 2098, 3147 and 3148 are open on your network. The following command can run from the OneFS command line interface on either the source or the target cluster:
# telnet
Can you also let us know a little more about your sync policies? One example should be enough, but if you can provide a couple that would be helpful:
# isi sync policies list --verbose
From the output of this command, be sure to include a policy that frequently or may always fail in this manner.
For additional information regarding SyncIQ, refer to out SyncIQ Information Hub!
SyncIQ - Isilon Info Hub
Please let us know if there is anything else we can do for you.