Unsolved
This post is more than 5 years old
3 Posts
0
6633
February 25th, 2017 09:00
OneFS 8.0.1.0 - Adding new ADS provider doesn't use correct source interface to connect to DC
Hey all, I'm looking for some ideas or advice on what we could be doing wrong here:
We're having an issue adding a new ADS provider to a groupnet for a isolated tenant. We've confirmed that there is connectivity between the DC/DNS server and the nodes in the groupnet smartconnect pool.
PROD-ISI-02-14# ping prod.company.com
PING prod.company.com (10.1.208.5): 56 data bytes
64 bytes from 10.1.208.5: icmp_seq=0 ttl=128 time=0.546 ms
64 bytes from 10.1.208.5: icmp_seq=1 ttl=128 time=0.447 ms
64 bytes from 10.1.208.5: icmp_seq=2 ttl=128 time=0.469 ms
^C
--- prod.companny.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.447/0.487/0.546/0.042 ms
PROD-ISI-02-14# isi auth ads create --groupnet=Production --user=mike --verbose prod.company.com
password:
CLI timeout exceeded while waiting for the server to respond; the request still may have completed. Use the --timeout option (isi --timeout= ) to adjust the timeout and try again.
PROD-ISI-02-14#
It looks like the Join Domain process isn't using the DNS for the selected groupnet, and is trying to route from interfaces from other groupnets to find the DC. I looked through all the command line options and couldn't find a way to tell it exactly which interface it should be using. If I look at the firewall between the new tenant and the other groupnets, I see node IP's from other GN's trying to reach the new tenants DC anytime I try to join.
When trying to join via the WebAdmin, I get the following message:
Active directory provider was not joined.
The Active directory provider did not create due to the following error:
Could not locate a domain controller in domain 'PROD.COMPANY.COM'. Check that your domain name is correct and DNS is configured properly. Unable to join.: NERR_DCNotFound
Any ideas what I could be doing wrong?
Cheers
sjones51
252 Posts
0
February 27th, 2017 10:00
HI mknox,
It sounds like you might have an asymmetric routing issue. The only thing throwing me off with that theory is the fact that you can ping the name. Check out the output of <#netstat -nr> and see what the default route is (This output is on a per node basis). <#ifconfig> can be helpful in matching up interfaces with IP addresses. If it is the "wrong" interface, you could fix the issue with either static routes or source based routing. If you need a hand with it, Isilon Support can assist. https://onlinesupport.emc.com/SRCreate
This post gives a great explanation of the basic traffic routing on the cluster:
Routing from A to B on Isilon
https://community.emc.com/community/products/isilon/blog/2014/11/28/routing-and-isilon-how-to-get-from-a-to-b-and-back-again
sleef
3 Posts
1
February 27th, 2017 12:00
There is actually an issue in 8.0.1.0 where PAPI will not use the proper DNS servers defined by the groupnets. The fix is in the upcoming 8.0.1.1 (184268). But, Groupnets were not designed to segregate nodes within a cluster (ex: nodes a,b,c in groupnet0 and nodes z,y,z in groupnet1), but ONLY to segregate DNS for AD environments.
Do make sure that the node you are using has networking AND DNS access to the AD you are attempting to join. The workaround with your existing config may require a 'root' DNS that serves both/all DNS AD domains.
RobChang-Isilon
136 Posts
0
February 27th, 2017 13:00
I wondering if you could solve this issue by using an access zone dedicated to this tenant.
Using the access zone method, you can create a groupnet with its own ADS, gateway, and DNS. And use a SmartConnect IP that is dedicated to this zone,
mknox1
3 Posts
0
February 27th, 2017 14:00
Hey Rob,
That's exactly the setup we have. This tenant has a dedicated AZ, with a dedicated groupnet, with all the correct network settings and it's on SmartConnect. The issue is when we try to add the authentication provider to this groupnet, it is using the DNS/Node IP's from other Groupnets to try and resolve and join the domain.
mknox1
3 Posts
0
February 27th, 2017 14:00
Aha, this might be exactly what's causing this problem. Any idea when 8.0.1.1 might be going GA?
What you described is what we'll be doing, we don't segregate nodes per Groupnet, we'd like all our externally-networked nodes to have interfaces for each tenant, with their own authentication realms, smartconnects, shares/exports, etc.
Prior to upgrading to OneFS 8 we actually did that workaround to make this work; we built a DNS forwarder for each separate domain on the cluster. It worked great, but I'm really hoping to be able to do anyway with all that, it was one of the major selling points for OneFS 8 for us : )
sleef
3 Posts
0
February 27th, 2017 14:00
8.0.1.1 is expected to GA mid-march...but the fix for that issue may not necessarily solve this problem. That issue is tailored to the PAPI interface when using the '--node-dc-affinity' option to choose a specific DC, which results in the incorrect DNS to be used.
Although, we have tested this scenario with the necessary fix, and was only possible to join multiple ADs with segregated DNS between groupnets on a node that was a member of ALL groupnets - again, so long as that requirement is met, it should work.
If the above configuration is currently in place, then that bug/fix may apply here, as the impact of that is the default groupnet DNS (/etc/resolv.conf) will be used exclusively for DNS lookups for AD realms/domains. If the above configuration is *not* in place, then that is a basic requirement for this to work (technically, all nodes in all groupnets...).
The workaround would be to include a "root" DNS in the default groupnet (groupnet0) which will populate /etc/resolv.conf and should work just fine.
The secondary issue with this setup is networking/routing. You may need to configure static routes for each DNS server to point out the correct Gateway for each subnet/gateway (one rule per node per subnet; no need for a rule for each pool). On top of that, you may also need static routes for each AD DC if they are not on a local subnet to the cluster, to avoid the default gateway for each DC (my guess is you might need this when a network setup gets more complicated than more than 2 subnets per node).
Bear in mind, we are a single server with a single default gateway, and can only assume the correct route is the default for non-local subnets/hosts.
I may have mis-interpreted the ask here, as the title suggests the latter, and the body of the post suggests the former, which might be leading to some confusion.