Avamar hardware documentation

All Avamar Gen4S nodes have 8x NICs running at 1 Gigabit.

ok, that's what we are planning to deploy, small Avamar Grid and DD890.  So if i have 3 storage nodes, with 6 NICs available per node, that's 18 NICs that i could possibly connect. Does that make any sense ? Le'ts say i connect 2 NICs from each node for a total of 6 NICs, i assume each NICs will have it's own IP address. How does Avamar manage/load-balance the traffic, the agent only knows avamar by one DNS name ?

Sorry for all the questions and thank you

For multi-node grids, eth1 and eth3 are reserved for the internal network.

Configuring LACP would require an RPQ for sure. The only redundant network connectivity supported out of the box is active-backup. I have yet to see a customer implement LACP because most often there is just no reason to do it. The network is very rarely the bottleneck for Avamar.

• Typical client filesystem I/O performance is in the 100GB/hr range
• The Avamar server can typically ingest about 30GB/hr per node (75GB/hr per node for 7.8TB nodes)
• An Ethernet network can support about 350GB/hr. Avamar typically compresses data at a 2:1 ratio so that gives us roughly 700GB/hr of network throughput per node in the default configuration.

Network bandwidth on the Avamar server becomes even less important if your data will be stored on the Data Domain back-end. The amount of data sent to the Avamar server in an Avamar / DD integrated solution is quite low.

got it, thank you Ian.

Do you see a lot of customers doing LACP with storage nodes ? I have Cisco Nexus 5k so i can do LACP and connect 4 NICs to one switch and 4 NICs to another switch ? Can i form a 8 port LACP port channel on Avamar node ?

Thank you

can i configure multiple VLANs on bond0 ?

Yes, absolutely. This is best done at install time if possible but professional services has a procedure for adding VLANs post-install.

How would that work with utility node being my "load-balancer" per say ?

To clarify, it's not really the utility node acting as the load balancer -- the utility node doesn't know how busy the other nodes are, it only has a list of their IPs. In fact, when a client connects to a storage node, it saves a copy of the list of storage nodes (called the "dispatcher table") and bypasses the utility node completely for subsequent backups. It will fall back to the utility node if it can't reach any of the storage nodes in the dispatcher table.

In any case, once the VLANs are configured at the OS level, the interfaces have to be added to the Avamar "probe" file so the Avamar software is aware of them. There will be one interface entry per IP address, essentially. Whenever a client connects to the server, it sends the utility node IP address it's trying to connect to (the "key address"). The server looks up the key address in the interface list and sends back storage node IPs from the corresponding network group.

i see, makes sense. So if there is no load-balancer and utility node simply has a list of IP addresses of each node (for each VLAN). How does it decided which IP address to give from the interface list ? Round-robin or something more sophisticated. If i have 800 server registered in Avamar, i don't want 500 of them to gang up on node 1.

The storage nodes themselves do the load balancing. There are three connections that happen if no dispatcher table is available on the client:

Connection 1

1. Client connects to ascd on the utility node
2. ascd replies with a random storage node IP
3. Client disconnects

Connection 2

1. Client connects to random storage node specified by ascd
2. Storage node replies with IP of least busy node
3. Client disconnects

Connection 3

1. Client connects to least busy node and begins sending backup data

These first two connections are very short and hit nodes at random. If there's a dispatcher table on the client, it will skip the first connection and just pick a random storage node to connect to. The third connection for load balancing will still be done.

The client is smart about not ganging up on one node.

The standard configuration looks like this:

eth0 and eth2 bonded as active-passive bond0. eth0 from each node connected to customer switch A. For an (optional) HA configuration, eth2 would be connected to customer switch B. The IP address for each node is assigned to bond0. The bonded IP fails over from eth0 to eth2 in case of issues with eth0.

eth1 and eth3 bonded as active-passive bond1. eth1 from each node connected to Avamar Communication Module A and eth3 from each node connected to Avamar Communication Module B. By default, bond1 is assigned an IP from the 192.168.255.0/24 network. This network is only reachable from the Avamar nodes. The bonded IP fails over from eth1 to eth3 in case of issues with eth1.

The additional NICs on the nodes are intended to support configurations like having a dedicated replication network between grids. The source system would have one of the unused NICs on the utility node connected to the replication network and assigned an appropriate IP. On the target, we would need one NIC on each node for this. In this type of scenario, we normally set up static routing on the Avamar itself to ensure that traffic for the dedicated replication network gets to where it needs to go.

Traffic is never load balanced between NICs in Avamar, only between nodes.

Whenever a client connects to the utility node, a process called ascd redirects the client to one of the storage nodes. The client disconnects from the utility node and connects to the specified storage node. The storage node gives the client the address of the least busy node and the client then disconnects and reconnects to the least busy node.

The data streaming to the server isn't necessarily written to the storage node that receives it. The receiving node just acts like a dispatcher -- any data that needs to be written to a different storage node will be sent to that node across the back-end network.

Long story short -- if you want front-end HA, you need to connect two NICs from each node, one to each switch. If you don't want HA, you only need one NIC per node. Additional NICs don't get you anything unless you're trying to do some kind of traffic segregation.

Edit: Clarity

good stuff, thank you !

can i configure multiple VLANs on bond0 ? We have many different networks that are separated by firewalls, i realize that Avamar does source based dedupe and all that good stuff ..but i would prefer to have Amavar connected to different networks and keep backup/metadata traffic local to that specific LAN. Is that doable ?  How would that work with utility node being my "load-balancer" per say ?

makes sense, very nice.

I still have this one question in my mind about ascd and VLAN tagging. Let's say this is my client network:

DMZ - 10.224.144.0/24

Admin - 160.140.144.0/24

HIPAA - 140.140.144.0/24

Here is my Avamar grid config

Utility node

bond0.vlan1 (DMZ) - 10.250.144.20

bond0.vlan2 (Admin) - 165.140.144.20

bond0.vlan3 (HIPAA) - 145.140.144.20

Storage node 1

bond0.vlan1 (DMZ) - 10.250.144.21

bond0.vlan2 (Admin) - 165.140.144.21

bond0.vlan3 (HIPAA) - 145.140.144.21

Storage node 2

bond0.vlan1 (DMZ) - 10.250.144.22

bond0.vlan2 (Admin) - 165.140.144.22

bond0.vlan3 (HIPAA) - 145.140.144.22

Storage node 3

bond0.vlan1 (DMZ) - 10.250.144.23

bond0.vlan2 (Admin) - 165.140.144.23

bond0.vlan3 (HIPAA) - 145.140.144.23

let's say i have a client on DMZ network (10.224.144.0/24), when i install agent on that client i will need to specify IP address 10.250.144.20 as my Avamar server correct ? Let's assume this is a brand new client, client connects to utility node and ascd should give it a random IP address of storage node.  How will it know what IP address to give it back to the client, to ensure it's on the same network "core" (DMZ) in this case ?

Thank you so much Ian.

When the client connects to ascd, it provides the server with the utility node address it's trying to use (called the "key address"). In probe.xml (the node database), the network interface definitions have an attribute called "nwgrp". The network group for each interface in a VLAN will be the same, so 10.250.144.20, 10.250.144.21, 10.250.144.22 and 10.250.144.23 might be assigned nwgrp="1", for example. These network groups are internal to Avamar so they don't need to match VLAN tags or anything like that.

The key address for the DMZ VLAN in your example would be 10.250.144.20. When the client sends that key address to the server, the server knows to respond with an IP address from network group 1.

My pleasure!

very slick, i like it.  Once again thank you very much Ian !

RPQ yeah ..those can get fun.

So the RMM interface ..in shared mode it's just functioning as a console port but i won't have remote power management, where with dedicated more i will have my virtual "power" button in addition to console ?

Thanks

oh darn, thought of something else

is active/active LACP not supported ? Also do you think i would ever need "console" connectivity to a node(s) ? Are there power management options (think of Dell iDRAC) for utility/storage node. Let's say utility node locks up and need to "push" the button.

You're very welcome!