This article will talk about Avamar SSH Key.

We know that after logging to Avamar, we need to run "ssh-agent bash", and then run the "ssh-add ~ / .ssh / dpnid" command to add the Key. Then we can login to any storage node without a password. So, do you know how Avamar uses the Key?

Before talking about the Key, we need to know what the Key is and does it work when using SSH to do a remote login.

The Key is used to encrypt and decrypt. Like a lock, we use the key to lock it, and then use the key to unlock it. For example, after encrypting a string of characters, it will become a pile of characters that cannot be recognized or a garbled code. So, if there is no key, you cannot decrypt the encryption and cannot know the specific content. There are two commonly used keys: a private key and a public key.

Both symmetric and asymmetric encryption use a private key. With symmetric encryption, the private key is used for both encryption and decryption. With asymmetric encryption, the public key is used for encryption and the private key is used for decryption. Encrypted by public key must be decrypted with the corresponding private key, and the converse is also true.

Here's a schematic diagram of the symmetric encryption (encryption and decryption are using the same Key):

Here is a schematic diagram of the asymmetric encryption:

(Encrypted by private key, decrypted by Public key; and encrypted by public key, decrypted by private key)

Next, we will introduce the theory of SSH Key. SSH is a protocol that helping us to use PuTTY or other tools to remote login to server. Normally we use the SSH command to log into the remote server. Every time we want to log on, we need to manually enter the user ID and password. On Avamar, if we want to log in to the Storage Node from the Utility Node, we need to enter the user ID and password every time. It is very painful and inconvenient. So we figure out how to login to SSH without a password through Avamar's key functionality.

How to login on SSH based on the public key and private key encryption technology to do an automatic certification.

This is usually done through the ssh-keygen command to create the authentication key. A public key must be placed on the server (adding it to the file ~/.ssh/authorized_keys), and a private key, which corresponds, to the public key in the client's ~/.ssh directory. The public and private keys are extremely important.

Let us look at the following chart for SSH login without a password process:

Avamar uses the golden key to confirm the login between the Utility Node and the Storage Node. So, does Avamar use a conventional SSH command? The answer of course it is not. Avamar have its own login method.

Approach 1: SSH-agent bash

This command should be used first to activate the other command functions. This command will save the private key which needs decryption.

Approach 2: ssh-add ~/.ssh/dpnid

Put the private key of the DPN account into the cache, it will automatically get the private key.

Approach 3: ssh-add -l

This is used to list all currently loaded keys.

Approach 4: ssn 0.*

We can use the SSN command on the utility node to log on to any storage node. 0.* is storage node ID number, such as 0.0, 0.2 nodes. Since Avamar can only have a maximum of 16 nodes, so that 11th node ID be 0.A, and the maximum node ID number is 0.F.

Since we can use the SSN to log on to any node, we can view which files relate to SSH on the Avamar node.

·         /home/admin/.ssh：

-r-------- 1 admin admin 744 May 31 2011 admin-id-2.dsa        

-r-------- 1 admin admin951 May 31 2011 admin_key               

-r--r--r-- 1 admin admin 223 May 31 2011 admin_key.pub      

-r-------- 1 admin admin823 May 31 2011 authorized_keys2 

-r-------- 1 admin admin27 May 31 2011 config        

-r-------- 1 admin admin668 May 31 2011 dpnid                 

lrwxrwxrwx 1 admin admin 9 May 31 2011 id_rsa -> admin_key        

·         /home/dpn/.ssh:

-r-------- 1 dpn admin 0 May 31 2011 authorized_keys2           

-r-------- 1 dpnadmin 27 May 31 2011 config          

-r-------- 1 dpnadmin 668 May 31 2011 dpnid                 

-r--r--r-- 1 dpnadmin 600 May 31 2011 dpn_key.pub           

lrwxrwxrwx 1 dpn admin 5 May 31 2011 id_rsa-> dpnid     

·         /root/.ssh:

-r-------- 1 root root 1209 May 31 2011 authorized_keys2   

-r-------- 1 root root 27 May 31 2011 config 

Next, we will look at the private key. As you can see we simply cannot read the contents.

admin@gen4-util:~/>: cat /home/admin/.ssh/dpnid

-----BEGIN DSA PRIVATE KEY-----

MIIBuwIBAAKBgQCWUMSv1kpW6ekyej2CaRNn4uX0YJ1xbzp7s0xXgevU+x5GueQS

mS+Y+DCvN7ea2MOupF9n77I2qVaLuCTZo1bUDWgHFAzc8BIRuxSa0/U9cVUxGA+u

+BkpuepaWGW4Vz5eHIbtCuffZXlRNcTDNrqDrJfKSgZW2EjBNB7vCgb1UwIVANlk

FYwGnfrXgyXiehj0V8p9Mut3AoGANktxdMoUnER7lVH1heIMq6lACWOfdbltEdwa

/Q7OeuZEY434C00AUsP2q6f9bYRCdOQUeSC5hEeqb7vgOe/3HN02GRH7sPZjfWHR

/snADZsWvz0TZQuybs8dEdGh/ezGhiItCINFkVg7NvSXx85dMVsB5N9Ju0gDsZxW

/d41VXYCgYBH0zIlb3lvioedyZj2mKF6fycnCZIeeDnL8wZtZPStRht6i4PFTCX1

Y/Ogw0L0bhuthOx+VTgICB87r0TmXElNUDLSncsxuw7pmHa669idUkv43CjeDkH0

kGFEHt4QA6/xw1Xq9oNpRJTo62ZsFmv0Pwp3uE7up8s0LW1O6fr+OwIVAKCJZ8nm

UwIdhEc9aU7sBDTFijP+

-----END DSA PRIVATE KEY-----

admin@gen4-util:~/>: cat /home/admin/.ssh/admin_key

-----BEGIN RSA PRIVATE KEY-----

Proc-Type: 4,ENCRYPTED

DEK-Info: DES-EDE3-CBC,DFDBC4B8EF3708FC

f2/1GJu45AtuQBlhokUib9T3m4o54cehWHx5z/BdbUYpTqsf7420/Dsb2o69ALR0

....................................

TcPSa3ARIpbuoPzOkR4osl+gzZzTaSHXUTbMPh79d3E=

-----END RSA PRIVATE KEY-----

Now we take a look at the public key:

admin@gen4-util:~/>: cat /home/admin/.ssh/authorized_keys2

a.       ssh-rsaAAAAB3NzaC1yc2EAAAABIwAAAIEAsf14EqXZb6MtXwAbou+UWEBU0Rj3ZBwaJUzSAHgPdWj35wK9l/z7mUfw4hzAmHJ+v/AOSZ++WDg9zi9JrpiSUdbkOKcoWPjEqbYjGMYbwyyIkUaNevY3sFgkh/+46lh/DYSJqVcbeUo1rSoUCJ03T5BtXQo/KEp/O27fHyry3FU= dpn_admin_key àà admin_key.pub

b.       ssh-dssAAAAB3NzaC1kc3MAAACBAJZQxK/WSlbp6TJ6PYJpE2fi5fRgnXFvOnuzTFeB69T7Hka55BKZL5j4MK83t5rYw66kX2fvsjapVou4JNmjVtQNaAcUDNzwEhG7FJrT9T1xVTEYD674GSm56lpYZbhXPl4chu0K599leVE1xMM2uoOsl8pKBlbYSME0Hu8KBvVTAAAAFQDZZBWMBp3614Ml4noY9FfKfTLrdwAAAIA2S3F0yhScRHuVUfWF4gyrqUAJY591uW0R3Br9Ds565kRjjfgLTQBSw/arp/1thEJ05BR5ILmER6pvu+A57/cc3TYZEfuw9mN9YdH+ycANmxa/PRNlC7Juzx0R0aH97MaGIi0Ig0WRWDs29JfHzl0xWwHk30m7SAOxnFb93jVVdgAAAIBH0zIlb3lvioedyZj2mKF6fycnCZIeeDnL8wZtZPStRht6i4PFTCX1Y/Ogw0L0bhuthOx+VTgICB87r0TmXElNUDLSncsxuw7pmHa669idUkv43CjeDkH0kGFEHt4QA6/xw1Xq9oNpRJTo62ZsFmv0Pwp3uE7up8s0LW1O6fr+Ow== dpn@dpn41s àà dpn_key.pub.pub

We can see the public key is placed in the authorized files.

So, how do we check whether the key is matching and correct? Of course we have the approach.

Approach 5: ssh-kengen –y -f

Use this to see whether the public key is same with the real public key. For example, we use this command to figure out the public key, and then compare with the real public key. If its the same, it means it's a match, otherwise, it does not match.

admin@gen4-util:~/>: ssh-keygen -y -f /home/admin/.ssh/dpnid

ssh-dssAAAAB3NzaC1kc3MAAACBAJZQxK/WSlbp6TJ6PYJpE2fi5fRgnXFvOnuzTFeB69T7Hka55BKZL5j4MK83t5rYw66kX2fvsjapVou4JNmjVtQNaAcUDNzwEhG7FJrT9T1xVTEYD674GSm56lpYZbhXPl4chu0K599leVE1xMM2uoOsl8pKBlbYSME0Hu8KBvVTAAAAFQDZZBWMBp3614Ml4noY9FfKfTLrdwAAAIA2S3F0yhScRHuVUfWF4gyrqUAJY591uW0R3Br9Ds565kRjjfgLTQBSw/arp/1thEJ05BR5ILmER6pvu+A57/cc3TYZEfuw9mN9YdH+ycANmxa/PRNlC7Juzx0R0aH97MaGIi0Ig0WRWDs29JfHzl0xWwHk30m7SAOxnFb93jVVdgAAAIBH0zIlb3lvioedyZj2mKF6fycnCZIeeDnL8wZtZPStRht6i4PFTCX1Y/Ogw0L0bhuthOx+VTgICB87r0TmXElNUDLSncsxuw7pmHa669idUkv43CjeDkH0kGFEHt4QA6/xw1Xq9oNpRJTo62ZsFmv0Pwp3uE7up8s0LW1O6fr+Ow==

root@gen4-util:~/>: cat /home/dpn/.ssh/dpn_key.pub

ssh-dssAAAAB3NzaC1kc3MAAACBAJZQxK/WSlbp6TJ6PYJpE2fi5fRgnXFvOnuzTFeB69T7Hka55BKZL5j4MK83t5rYw66kX2fvsjapVou4JNmjVtQNaAcUDNzwEhG7FJrT9T1xVTEYD674GSm56lpYZbhXPl4chu0K599leVE1xMM2uoOsl8pKBlbYSME0Hu8KBvVTAAAAFQDZZBWMBp3614Ml4noY9FfKfTLrdwAAAIA2S3F0yhScRHuVUfWF4gyrqUAJY591uW0R3Br9Ds565kRjjfgLTQBSw/arp/1thEJ05BR5ILmER6pvu+A57/cc3TYZEfuw9mN9YdH+ycANmxa/PRNlC7Juzx0R0aH97MaGIi0Ig0WRWDs29JfHzl0xWwHk30m7SAOxnFb93jVVdgAAAIBH0zIlb3lvioedyZj2mKF6fycnCZIeeDnL8wZtZPStRht6i4PFTCX1Y/Ogw0L0bhuthOx+VTgICB87r0TmXElNUDLSncsxuw7pmHa669idUkv43CjeDkH0kGFEHt4QA6/xw1Xq9oNpRJTo62ZsFmv0Pwp3uE7up8s0LW1O6fr+Ow== dpn@dpn41s

Then we log on to a storage node, we will find that all the machine through SSH to login successfully, their public key will be in known_lists for each account. Because there are three Avamar system account: admin, DPN and root. So, as long as the account that is used to SSH login, known_lists is in this account there will be a public key for this. 

account:admin@gen4-data1:~/.ssh/>: cat known_hosts

1.       10.32.167.88 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA49Ueggi8WrdvTP+1cenYNSSrPv8GMJyiFqcsmDyqOQuQiLxiirtORm7P633JZrW4vzpHQZzdd0Q4ASybYXbaHB2F+S3UMl9x9OGaWigk06SgWLF6Kh9kbjFgFYCIIpZW1RTo75N7r7f+rUmLCCmrYYi3Cy1XhEEMedKB948WnG0=

2.       10.32.167.84 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAsV1RM7h5RDT7EfBktWbU0Ezm96QlHYZoq6JbbMJkThzleOzX+GzO5qqIBXUiDvIP08jSNuDlfCO9aTEQivYmZZoHpYdsPpxW/YZIw0a4+Pa+mzwqFkmwwZGe+ck/nUjJ+3tfpiDtNbMBg+cAJGrrkUyo4drL7SW0D5qfg5cU/0c=

3.       192.168.255.1 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA74EBveMGzDU5laPV2vBhaAODFOPHUO33hacBIzotX+mBnyq+jhBBuuFU12PstkBleUUWOz6JrefT/HPaTXojyzqQWdVD+j97RZg9EZY3JqwRjFrcjEZ89JfNY43T7lJGx8tKLAt2HB2gg9CU82QLvOOdgTuewh6736G3CVu1LI0=

4.       10.32.167.135 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA7mZB7Eksd6S2SZGbm/yp/caQsSKi+bUItDog4uh12GuT5oxe6hFZtM2gJEMlGQvbqQMw+gMpUckrrYINWB7iaNXgzdK08YCd4/EUP8E74WvbG2h3WCqsYJ8FoP2qR3DXPzzoUQkwyNoTsGL3NDcVqRnylYDarRyWlHCfXCVNJNM=

5.       10.32.167.99 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA74EBveMGzDU5laPV2vBhaAODFOPHUO33hacBIzotX+mBnyq+jhBBuuFU12PstkBleUUWOz6JrefT/HPaTXojyzqQWdVD+j97RZg9EZY3JqwRjFrcjEZ89JfNY43T7lJGx8tKLAt2HB2gg9CU82QLvOOdgTuewh6736G3CVu1LI0=

6.       gen4-util ssh-rsa

AAAAB3NzaC1yc2EAAAABIwAAAIEA74EBveMGzDU5laPV2vBhaAODFOPHUO33hacBIzotX+mBnyq+jhBBuuFU12PstkBleUUWOz6Jr

As the example above, we can find that the storage node's admin account including multiple public keys which are Utility Node and other nodes.

Approach 6: ssn --user=root 0.<node ID>

We can use the root account to log on to any node. As in the following example:

Approach 7: mapall

This command can help us to list information on multiple nodes simultaneously. But need ssh-agent bash and SSH ~/.ssh/dpnidto load the key and through SSH to use it.

To use this command, we can also list all nodes online days through SSH. Of course, this command can also be used with other parameters.

Today we understand the SSH key, right? Next time, we will introduce the use of SSH Key on the Data Domain and how to troubleshoot SSH Key error issues.