How does SmartConnect DNS work? And SIP details?

You have the flow down well.  The only servers who should be talking to the SIP are DNS servers. Clients should not be talking to that address.  Any client who does would just connect to the node that is running the SmartConnect service (usually the node with the lowest LNN with a network connection on that subnet).

There really isn't a specific management IP unless you define one.  It's just not something officially defined in OneFS.

With respect to the placement of a SIP, what really matters is where the DNS servers live.  Each subnet *can* have a SIP but it's not required.  Each IP Pool needs to have a SmartConnect subnet defined, but that can be it's own subnet or another one.  So it's entirely possible to have one SIP for the entire cluster even if that cluster has multiple subnets that don't route to each other.  As long as the DNS servers that need to talk to the cluster can do so, that's all that is required.  A SIP can hand out addresses from any IP pool to which is is assigned.  So if you have a "management" subnet and that's where the DNS servers are connected, then the SIP on that management subnet can server IPs for any subnet on the cluster.  Whether you need multiple SIPs or not really depends on your network design.  The cluster can accommodate most designs these days, especially in 8.0+

Hope this helps.  There's a lot wrapped up in those questions, but I figured I'd give it a first shot.

I am really hesitant about having another "DNS" device on our network. I am on the security team and not directly involved with the product installation from the "use" standpoint. Is this all mandatory to have implemented?

It's not really a DNS "device". It is only authoritative for itself, meaning the SmartConnect zone names that you delegate to it in DNS.  But if you want the cluster to function properly, it has to make the decisions about where to send the traffic, based upon which nodes are up, the load on each node, etc.  It also then has the ability to control failover.  I get that the first time many customers see SmartConnect they get very confused, and from time to time there is some resistance, but I've never seen a customer not implement it in the end. 

I would strongly suggest you read this guide:

EMC Isilon External Network Connectivity Guide: Routing, Network Topologies, and Best Practices for SmartConnect

Disclaimer; I did write a substantial amount of it.

~Chris

It's not mandatory, but if you don't use the SIP, you will be required to manually balance the front-end client connections on your cluster.  How easy this is will depend on the # of clients connecting to your cluster.   If it helps, this is not a fully implemented DNS server.  It will only answer simple A record lookups for the names of the SmartConnect zones so there's not much it can do from a security standpoint.  You could always put the SIP on a private VLAN that only your DNS servers can reach if you're really concerned.  Bad actors can't exploit what they can't reach.

Is\can the SIP IP be different than the Isilon management IP?