NetWorker sends DMZ client to a TCP port out of configured range

Thank you Bill,

A colleague was working on this before I took over and they did try to calculate the required number of ports.  Exactly how relevant is it to calculate ports for a backup client?  I understand it's important for a server or a storage node.

EDIT: I've just re-read your post.  I'll look in to doing the calculation, thanks!

My client was the one concerned about opening too many TCP ports, so thank you for clarifying that it doesn't compromise security.

I just set the server down to the same range, here's my check:

C:\Users\(admin-user)>nsrports -s (backup server)
Service ports: 7937-7957
Connection ports: 0-0

but it still replies to the client's portmap request with a port out of the range!

GETPORT Reply (Call in 112) Port:9879

What should I try next please?

Take 2!

Bill, could you please clarify this for me?

"
Setting an arbitrary range of 21 which has not been calculated for the environment is not of much use really.

You can reduce this range to 4 ports for the NetWorker server to client communication, however without calculating and setting the range the Networker server can use you will always have these issues.
"

I'm not sure how the number of 21 ports were reached, but I'm still a little confused as to why these problems would occur:

The way I read the "Configuring TCP Networks and Network Firewalls for EMC NetWorker" document I thought that if clients had sufficient ports for their own communication, that would be enough for backups.  Are you saying that the client-server communication needs enough ports for the server's own communication please?

That is, does the server need all the extra ports used for communicating with various devices and storage nodes [using the 12 + (2 * #devices) + (#jukeboxes) service ports formula] just to communicate with a single client for backup?

Surely between the server and client it should only need the ports required for client communication?

Thank you in advance!

Dave,

Setting an arbitrary range of 21 which has not been calculated for the environment is not of much use really.

You can reduce this range to 4 ports for the NetWorker server to client communication, however without calculating and setting the range the Networker server can use you will always have these issues.

Setting the range for Service ports will not have any effect on LAN backups as the NetWorker daemons will start within this range and be managed accordingly - this range is not used for anything other than daemon process communication/authentication.

There is no security risk posed by having a rule for 21 ports as now or having rule for 50 ports, having a range of 4 ports is a rule does not make your environment more secure. This is a common misconception with Customers but once explained they usually allow the rules and ranges to be configured.

Regards,

Bill Mason

Dave,

Taking your points: -

1. Port range of 21 - currently you said the firewall rule was configured with a range of 7937-7957 ( which is 21 ports).

2. The NetWorker server range on the server is set to 7937-9937 (2001 ports). So the NetWorker server can select ports from the larger range, of which anything over 7957 will be "barred" by the firewall.

3. If you change the range on the NetWorker server, you must restart the software to pick up this amendment.

4. When the NetWorker server is talking to the client it will talk to nsrexecd as this is the only daemon communications are directed to. Nsrexecd listens on ports 7937 and NetWorker portmapper is 7938 always. From NetWorker 7.3.x and above has 4 ports available 2 for nsrexecd and portmapper and 2 user configurable ports. So the range for communication from the NetWorker server and Storage node outside of the DMZ with the client port range usually set to 7937-7940 within the firewall rule from the NetWorker server/Storage node outside of the DMZ to any client within the DMZ as no version below 7.5.x is currently supported by EMC.

a. There are another two ports which you could add into the firewall rule as there is a fallback port which can be used if nsrexecd cannot be contacted for some reason which is 514. External portmapper could also be used if Networkers' is unavailable for some reason which is port 111. So you can configure these 2 ports also in the rule which points from NetWorker server/Storage node to any client in the DMZ, making a total of 6 ports in the range.

5. Once the Server has communicated with the client, the client then needs to reply to the NetWorker server and talks to nsrexecd, nsrd, nsrauth etc and then later nsrindexd and maybe to nsrmmd. Within NetWorker other daemons apart from nsrexecd (it is fixed along with Portmapper) can be started on any port within the range defined, so if you have range of 2001 ports available we can pick any number within this range. The process does not always start from 7939 and go sequentially upwards, as the OS decides which port and as long as its free and in the range we will use.

a. The document tells you how to work out the ranges for the Storage nodes and NetWorker server which would allow firewall rules to be set appropriately for client to NetWorker server and client to Storage node if storage node is outside of the DMZ or NetWorker server to Storage node if it is.

b. The storage node if within the DMZ can be added to the same rule configured for the client.

6. If you set the range too low NetWorker will have issues as all daemons may not be able to run, we allow for nsrmmd restarts within the calculation (as previous port may not have been freed up before we attempt to restart) and it is always good to allow a few extra in the calculation to give you some resilience to adding temporary devices etc. Significant system growth for Storage node, library addition etc will require the range to be recalculated and amended on the NetWorker server, Storage nodes and firewall rules, and of course a restart of the software.

Hope this helps.

Regards,

Bill Mason

Thank you for your lengthy reply Bill, I appreciate it.

I'm afraid I'm still quite confused!

• Could you please answer this for me: What is the point of setting service and connection port ranges on a backup client when the NetWorker server seems to ignore these and try to have the client use a port outside its range?!
  

My client's gateway provided is refusing them to open the default 2000 TCP port range use for NetWorker's service ports.  We've managed to get them to compromise on a 500 TCP port range, which I hope will easily be enough.

After I understand why NetWorker seems to ignore each client's port range configuration, I'll look in to the Connection ports.  I'd love to understand which OS (the NetWorker server/storage node or the backup client!) decides on the connection ports to use when "0-0" is specified.  The documentation does not explain this.

If I do understand everything correctly, I plan on allocating the first 200 TCP ports for service ports and the remaining 300 for connections.

Service Ports:          TCP 7937-8136

Connection Ports:    TCP 8137-8436

Thanks again,

Dave

Hi Dave,

Service ports are incoming ports while connection one are outgoing.  Usually, as far as DMZ goes, you are interested in protecting from incoming connections.  As seen in your first post, backup server can use wider range of ports to accept and send from service requests.  My understanding is that your client replies back to higher port on range side.  The problem there might be if you protect this on bidirectional level - in such case your will need adjustments on both sides.  I'm personally not big fan of blocking ports as 10 or 10000 ports open - if application is open all I need is 1 port to application.  Of course, there are certain DMZ situation where things are just close - in that case I prefer isolated solution for DMZ.  In your case, I would check with security if what they see affects incoming or outgoing traffic in respect to DMZ and if outgoing filtering is really necessary in this case (as they could easily create exception which is valid for your range between specific hosts or only in communication to backup server).

Hi Dave,

Will you have storage node in DMZ too?  Or will data go our of DMZ?  Anyway, to make it simple, let's assume backup server initiates backup and let's say you have 40 clients.  To make it simple and port friendly, I will assume you will use parallelism of 1.  In such case 7937-7940 is more than enough on client side.  Obviously server side needs more and for that you must take into account the whole datazone when calculating.  So, we assume server contacts client and reads NSRLA records where ports are define, does auth thingy and decides what to backup and save is ready to be spawned with data.  In this case data from client is pushed to storage node using connection ports (data transfer ports would be better name).  Unless you have changed this, your nsrla record (as seen by nsrports) still has 0-0 meaning use everything.  But that's source... and there is target too (port on which data on storage node will land).  Now, if sorage node is in DMZ then that's easy as no restrictions stand in your way.  Communication between backup server and storage node (data used for updating media database) and communication between backup server and client (data used for updating index database) uses service ports.  I didn't check current math and formulas, but I think it should be easy to calculate range and then add 100 ports on top of that just to be sure.  The downside with bidirectional filtering is that you must make changes on hosts like backup server which has impact then on whole datazone (and storage nodes too).  While I have couple of DMZ locations, we used mixed approach (depending on country, but we might soon consolidate this to one approach for all, but not sure which one).   In some instances we have port range open - I used traditional port range by NW for connection ports even that one is way bigger than needed.  network ppl and security accepted the fact that these ports are open only towards dedicated backup machines via only dedicated network so I didn't have problems.  In few other locations we have model of closed DMZ; this means no DNS, no AD, no nothing (now, that's real DMZ).  In such locations we did simple approach where we use ESX server and all machines are virtual (where they were not we did P2V) and backup is done at VmWare level.  Eventually, if have to use some dedicated backup software for it I would probably use small Avamar Virtual Edition.  But of course, I can understand your position and I know that you are stucked with what you have.  I'm not a code expert (nor programer in the first place) so I can't coment wether port ranges used by NW are something that can be optimezed or not.  I believe Avamar requires just one port which sound great while NW happily assigns 1 port per connection.  I guess there is probably some catch if you at TCP level try to push everything over single port and it does have some sort of impact so at first ball spread approach sound as better bet... no idea really.

My suggestion would be to check what port range you need and then increase it for 1000 for example.  Let sec&net people know this is just temporary and then as you put it into run you will decrease it and have monitored this to make sure it works.  I believe the issue you have is most probably connection port range which you will need to set to something acceptable.  Major challange will be the fact that this is bidirectional filtering as it means you will need to make changes also outside DMZ and this will have impact on other clients outside DMZ (even I do not see why and if security guys are smart enough explaining that outgoing ports in wide range towards only dedicated host(s) would probably work).  Your connection ports seem to be in range what traditionally is service port range so I would place those from 10001-whatever needed.

Thanks for your response Hrvoje.

I appreciate your time and effort!

Unfortunately your response, as a huge block of text, is difficult to read.

Also, I'm afraid you've gone off topic in your response.  We don't want a storage node in the DMZ, and we don't user Avamar.  We aren't using virtual platforms in the DMZ either.

I consider our DMZ setup relativley simple and common for any business that takes internet security seriously.

I can't just have firewall ports opened for testing like you suggest!  Security is a big deal here.

I could spend some time mucking around with a firewall I control (perhaps by using a software firewall) but I can't stop thinking it's a waste of time.  I expect EMC to supply clear documentation with their product, such that we don't have to use a packet sniffer and a test firewall just to discover how it works!

Unfortunately my issue (which really should be pretty basic) is still not resolved.

Dave-N,

I apologise for the delay in responding to you but I have been OOO on vacation. I just want to be sure of your settings which are in place, as this seems to cause confusion for a lot of people despite many attempts to make the documentation more clear.

Please note: -

1. All the comments below do not take into account Legato License Manager, NMC or other EMC. software which you could be using (details of all the other ports are in the document you quoted earlier).

2. I am also assuming the NetWorker server is the only Storage node you have so all client data will be directed to it, other wise we will need another firewall rule for communication from clients to the Storage node.

3. Connection port values for the NetWorker server and clients can be left at the default setting (0-0) as they do not require amendment or configuration on the firewall side.

For Communication from the NetWorker server to the clients within the DMZ the firewall rule should quote 7937-7940 as the port range. All clients can be named in this rule as no client will use a port outside of this range (unless it falls back to using the OS RPC portmapper on port 111 for some reason). So you could also include this port within the rule in case this situation occurs.

On each client the port range should be set to 7937-7940 only and the NetWorker software restarted to pick up the setting.

For communications from the clients in the DMZ to the NetWorker server the firewall rule should quote 7937-7976 for the 40 ports you say you have calculated as required for the NetWorker server (unless it falls back to using the OS RPC portmapper on port 111 for some reason). So you could also include this port within the rule in case this situation occurs.

The port range on the NetWorker server should be set to this range and the NetWorker software restarted to pick up the setting.

The only NetWorker daemons which have a fixed port number are nsrexecd and the NetWorker portmapper which start on 7937 and 7938 every time. All other daemons we start can vary but usually nsrd gets the next port available (7939) but it is not tied to this, it can use any free port in the range.

If the rules and nsrport ranges are set as described above on your Server/clients then I am confident your environment will work and produce successful communication and backups.

Regards,

Bill Mason

Dave-N wrote: Also, I'm afraid you've gone off topic in your response.  We don't want a storage node in the DMZ, and we don't user Avamar.  We aren't using virtual platforms in the DMZ either. I consider our DMZ setup relativley simple and common for any business that takes internet security seriously. I can't just have firewall ports opened for testing like you suggest!  Security is a big deal here. I could spend some time mucking around with a firewall I control (perhaps by using a software firewall) but I can't stop thinking it's a waste of time.  I expect EMC to supply clear documentation with their product, such that we don't have to use a packet sniffer and a test firewall just to discover how it works! Unfortunately my issue (which really should be pretty basic) is still not resolved.

Discussion about storage node and port has been added as so far you didn't define where storage node is.  Avamar has been mentioned as another approach in handling ports and that discussion has been expanded with what NW does - I never said you should consider that product (I was more likely hoping for someone else to jump the boat and have more details on what and why NW does the way it does).  I think your issue will be gone once communication port range is addressed.

Thanks again Bill,

I hope you had a good vacation!

1. At this stage there are only a few DMZ hosts and only filesystem backups are required (Windows & UNIX).
We do use other NetWorker backup modules within the LAN but it is unlikely these will be required in the DMZ.

I'm not sure how the Legato License Manager works, but am certain that it's not currently causing any problems with these DMZ backups.

2. Yes.  We do have a few storage nodes at remote sites, but the DMZ is located at our main site.  So the NetWorker server is also the storage node for DMZ clients.

3. Thank you for confirming that connection port values do not need changing.  I feel that the document doesn't really explain what they do, but now that you've told me I don't need to worry about them I won't!

For Communication from the NetWorker server to the clients within the DMZ the firewall rule should quote 7937-7940 as the port range.  All clients can be named in this rule as no client will use a port outside of this range (unless it falls back to using the OS RPC portmapper on port 111 for some reason).  So you could also include this port within the rule in case this situation occurs.

At this stage port 111 has not been opened, however we've not yet observed communication attempt to fall back to this port when doing packet analysis.

I had not realised that such a limited range of ports was only required outbound (LAN Server -> DMZ Client).  Thank you for clearing this up.

On each client the port range should be set to 7937-7940 only and the NetWorker software restarted to pick up the setting.

Ok, I had defined a larger range which the firewall was configured to allow through, but I can reconfigure this.

For communications from the clients in the DMZ to the NetWorker server the firewall rule should quote 7937-7976 for the 40 ports you say you have calculated as required for the NetWorker server (unless it falls back to using the OS RPC portmapper on port 111 for some reason).  So you could also include this port within the rule in case this situation occurs.

The last time I attempted a backup there were 200 *TCP* ports open (DMZ Client -> LAN server).  At that time the NetWorker server did not have its own ports restricted with 'nsrports' (The backup failed how I've described before in this thread).

The port range on the NetWorker server should be set to this range and the NetWorker software restarted to pick up the setting.

This is the bit that I find confusing.

I administer another backup product for another client;

• This other product allows me to configure restrictive port ranges for either specific backup clients (by name) OR for a range of IP addresses
  
  • (I also have to define the restrictive port ranges on each firewalled backup client.)
    
• The backup server using this product can continue to use the default, large, port range for any "normal" clients within the same network but knows to use the restrictive port range for specific clients only.
  

What concerns me about NetWorker is that by setting our NetWorker server up for communication with these DMZ hosts, we are affecting all of our LAN backups also.  I don't want to change what is already working.

So I've been trying to find a compromise with NetWorker;

• Few enough open ports so that our security/firewall people don't have a heart attack
  
• while ensuring there are enough ports for multiple simultaneous backups to multiple devices without loss of performance.
  

Now I gather that the idea of calculating the port range required is to avoid a loss of performance, right?  The thing is what if we add another DataDomain (and with it X-many devices)?  I'll have to get the port range increased and get another set of firewall rules done.

So, is there any way in which I can configure ports that affect DMZ backups ONLY please?

Thank you again!

David.

Thank you Hrvoje,

I appreciate your help, but understand that I work in an environments which may be quite different to your own.

Sadly, we don't have money available to purchase additional software or products (like Avamar) nor implement additional servers (e.g. Add a storage node to the DMZ).  We also have quite restrictive security requirements.

Yes, the problem is definitely related to firewall and NetWorker port configuration.

Dave-N,

Thanks for the reply, as you will appreciate backup products do perform the same basic function but can achieve this in a variety of different ways.

Restricting the range of ports available for the NetWorker server (or any storage node) does not have any effect on the performance of the software. The communication coming into the server is directed to the specific daemons required to perform the tasks, if the port they are using at the time is low or high numbered the process is not effected, the main thing is the communication can be completed.

Defining a smaller range or the default large range really makes no difference to the firewall rule. Firewall administrators always seem to complain if a large range is specified but as the ports are secure we always have issues getting this point across. Defining 40 instead of 2000 does not make the firewall rule/firewall or environment any more secure.

You are correct in stating that if more devices are added the range and firewall rule would have to be adjusted but only if the devices are actually increased on the NetWorker server itself, as the calculation now only takes into account the devices which are defined for the NetWorker server not all devices in the datazone. You would also have to consider any new libraries/jukeboxes created as this would again change the ports required if the NetWorker server controls the robotics.

Most people will build in some tolerance to this by defining a slightly larger range than required/calculated but you could end up with NetWorker wanting to start processes and have no ports available to do this if devices/libraries increase and the calculated port range is not adjusted along with the firewall rule.

As your environment evolves you may have to change the port range but this is not usually a big task or complex change as only a firewall rule amendment to the existing rule and nsrports command with software restart would adjust. Similarly if you add a new client in the DMX change firewall rule to add client into the list and set nsrports range on the client is all that is required. There is no need for a new set of rules to be produced, the existing rules can just be amended.

Defining the port range will not effect other backups within your datazone for non-DMZ clients, therefore there is no need to think of the backup process changing between the DMZ and non-DMZ clients, it is exactly the same for all we just need to go through the firewall for DMZ clients and have the correct rule/port ranges specified. Having a small range does not affect the number of clients which the NetWorker server can have defined as we don’t start daemons for each client. We just need to mange the processes are started to mange resources (devices, jukeboxes, index updates, media database etc), all other communications are via TCP/IP to nsrexecd on the clients or from the clients to specific daemons on the NetWorker server.

I hope this explains your queries, please let me know if you need anything further. I will take your comments and progress internally to see if we can improve the existing document to clarify further. It is common for incorrect perceptions/assumptions to be made, questions and issues with firewall administrators, but once we get passed the initial resistance and explain how the software works it is usually implemented and successful.

Regards,

Bill Mason

H,

I have never seen any firewall/DMZ restrict connection port ranges. I am by no means an expert in this field but from what I understand the only ports we need to be concerned with are the service ports.

I think the ranges previously specified are incorrect due to misunderstandings of the documentation/processes within NetWorker and not to specifically take connection ports into account.

Regards,

Bill Mason