SteveK821
2 Iron

Avamar Backups to DD with or without token authentication

I have a strange problem with AV/DD. Avamar Version 7.4.1-58

How does Avamar decide whether a client connects to the DD using token authentication or with credentials?

Some clients connect with token, but full backups (Lotus Notes) for these clients fail after a long time

Backup started on 2018-03-10 02:00:00 and starts OK.

2018-03-10 02:11:48 avtar Info <41236>: - Connecting to Data Domain Server name "mydd.mycompany.com" with token:ee0db7e0899dde98defb9b8df138cd92bbdd6029

2018-03-10 02:11:48 avtar Info <19156>: - Establishing a connection via token to the Data Domain system with encryption (Connection mode: A:3 E:2).

2018-03-10 02:11:48 avtar Info <0000>: - Connected to:

Data Domain System: mydd.mycompany.com

Model:              DD4200

DDOS:               Data Domain OS 5.7.5.6-580632

.

.

But then later on in the backup job it fails.

2018-03-11 04:39:36 avtar Info <41236>: - Connecting to Data Domain Server name "mydd.mycompany.com" with token:ee0db7e0899dde98defb9b8df138cd92bbdd6029

2018-03-11 04:39:36 avtar Info <19156>: - Establishing a connection via token to the Data Domain system with encryption (Connection mode: A:3 E:2).

2018-03-11 04:39:47 avtar Warning <18125>: Calling DDR_OPEN_VIA_TOKEN returned result code:5075 message:the user has insufficient access rights

2018-03-11 04:39:47 avtar Error <10542>: Data Domain server "mydd.mycompany.com" open failed DDR result code: 5075, desc: the user has insufficient access rights

2018-03-11 04:39:47 avtar Error <10509>: Problem logging into the DDR server:'', only GSAN communication was enabled.

2018-03-11 04:39:47 avtar FATAL <17964>: Backup is incomplete because file "/ddr_files.xml" is missing

Other backup jobs for the same clients (Log backups, Filesystem, Notes Incremental) work fine during the week.

Some clients in the same group and same domain whose full backups run at the same time work fine. After checking it I see that these clients that always work connect to the DD without a token, but with credentials.

Example from the Backup Logs:

Client connecting with Credentials

<directives>

    <flag type="string" value="tls" name="encrypt" />

    <flag type="string" value="high" name="encrypt-strength" />

    <flag type="string" value="1528416000" name="expire" />

    <flag type="string" value="daily,weekly" name="retention-type" />

    <flag type="string" value="avamar-ip" name="hfsaddr" />

    <flag type="string" value="27000" name="hfsport" />

    <flag type="string" value="backuponly" name="id" />

    <flag type="password" value="****" name="ap" />

    <flag type="string" value="/clients/client1.mycompany.com" name="path" />

</directives>

avtar Info <10539>: Connecting to Data Domain Server "mydd.mycompany.com"(2)  (LSU: avamar-1418052006, User: "********")

avtar Info <41234>: - Connecting to Data Domain Server name "mydd.mycompany.com" with credentials

Client connecting with Token

<directives>

    <flag type="string" value="tls" name="encrypt" />

    <flag type="string" value="high" name="encrypt-strength" />

    <flag type="string" value="1552179600" name="expire" />

    <flag type="string" value="daily,weekly,monthly,yearly" name="retention-type" />

    <flag type="string" value="avamar-ip" name="hfsaddr" />

    <flag type="string" value="27000" name="hfsport" />

    <flag type="password" value="****" name="ddr-auth-token-key" />

    <flag type="string" value="backuponly" name="id" />

    <flag type="password" value="****" name="ap" />

    <flag type="string" value="/clients/client2.mycompany.com" name="path" />

</directives>

avtar Info <10539>: Connecting to Data Domain Server "mydd.mycompany.com"(2)  (LSU: avamar-1418052006) with auth token

avtar Info <41236>: - Connecting to Data Domain Server name "mydd.mycompany.com" with token:ee0db7e0899dde98defb9b8df138cd92bbdd6029

Where does this difference come from? In mcserver.xml this is set:

<entry key="use_ddr_auth_token" value="true" />

<entry key="ddr_auth_token_duration" value="36000" />

As the duration is set to only 10 hours, this would explain the errors I am getting. I am planning on increasing the value of the duration and restarting MCS.

But it still does not explain why some clients connect without a token, even though they are using the exact same dataset. All other backups for those particular clients also connect using credentials. But I cannot find an avagent.cmd or avtar.cmd on those clients. I also don't see any overrides in Policy for those clients.

Any ideas? I have also opened an SR to see if support can shed some light on this.

Thanks and regards,

Stephen

0 Kudos
2 Replies
Highlighted
123kateb
1 Nickel

Re: Avamar Backups to DD with or without token authentication

We are experiencing same issue, in the last two weeks only (backups were fine up until then for this particular server).

Did you take a look at these, it this a configuration issue or software bug?  Please let us know what your solution was.

https://emcservice.force.com/CustomersPartners/kA2f1000000X3TyCAK

https://emcservice.force.com/CustomersPartners/articles/Break_Fix/Avamar-backup-to-Data-Domain-fails...

0 Kudos
SteveK821
2 Iron

Re: Avamar Backups to DD with or without token authentication

Hello Kate,

sorry, I was away on vacation over Easter. We fixed the issue by raising the value of ddr_auth_token_duration to 48 hours (172800 seconds). The full backups ran successfully, but I still never got an answer how Avamar decides which authentication method to use.

The problem was that the full backup session has several activities. But each activity uses the same token to authenticate with the DD. As some activities took a long time (over 24 hours in my case) the token had expired when a subsequent activity was started, causing the error message.

The other option would have been disabling token-based authentication in mcserver.xml

<entry key="use_ddr_auth_token" value="false" />

But we didn't need to do this. Increasing the token duration was enough.

Regards,

Stephen

0 Kudos