Avamar: Resolving Backups Stuck with Zero Progress (Session Ticket Login Error 3)

Summary: Avamar: Random Backups stop responding with Zero Progress (Session Ticket Login Error 3: MSG_ERR_INVALID_PARAMETERS)

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

VMware Image Jobs:

  • Random scheduled or on-demand VMware image jobs get stuck waiting for a secure progress message to connect to the Control (CTL) port.
  • Logs show that the job starts but makes no progress:
-------------------------------------------------------------------
-----  START avvcbimage <date>   [19.12.100-186 Linux-x86_64]
-------------------------------------------------------------------
avvcbimage Info <5008>: Logging to /usr/local/avamarclient/var/<logfile>-1016-vmimagel.log
avvcbimage Info <5174>: - Reading /usr/local/avamarclient/var/avvcbimage.cmd
avvcbimage Info <5174>: - Reading /usr/local/avamarclient/var/avvcbimageAll.cmd
avvcbimage Info <6673>: CTL listening on port 36461

    Logs show that the job starts but makes no progress:

2. Successful Backup on Restart:

  • If the backup is restarted using the same VM or proxy, it completes successfully.

 
3. Microsoft SQL Agent Based Backups:

  • Similar issues affect SQL agent-based backups:
 avsql Info <5174>: - Reading C:\Program Files\avs\var\avsql.cmd
 avsql Info <5612>: - - Reading CID 'C:\Program Files\avs\var\cid.bin'
 avsql Info <5614>: - - CID = 84827eed82c77f63aec5e1fecbbe5848ba7d6217 for "sqlclient" (1591293690)
 avsql Info <5008>: Logging to C:\Program Files\avs\var\logfile-3006-SQL.log
 avsql Info <6673>: CTL listening on port 64860
 avsql Info <10684>: Setting ctl message version to 3 (from 1)
 avsql Info <16136>: Setting ctl max message size to 268435456


4. Filesystem Backups (Windows/Linux) 

  • Filesystem backups may fail, not stop responding, with the following error:
avtar Info <5581>: Logging in on connection 0 with Session Ticket
avtar Info <18854>: Using Secure Session Ticket Format
avtar Error <8609>: Connection killed from GSAN.
avtar Info <9772>: Starting graceful (staged) termination, KILL event received (wrap-up stage)
avtar Error <5126>: Login error 3: MSG_ERR_INVALID_PARAMETERS (Session Ticket login)
avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
avtar Info <6149>: Error summary: 3 errors: 8941, 5126, 8609
avtar Info <8469>: Progress avtar, no wrapup.
avtar Info <5314>: Command failed (3 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
-------------------------------------------------------------------
-- END avtar log <date>  (0 warnings, 3 errors, 1 fatal error)
-------------------------------------------------------------------

 

Cause

The Avamar server is rejecting session tickets due to invalid parameters. The start time for the session ticket is incorrectly set in the future. The time difference between the Avamar Utility Node and the Avamar Storage node causes this discrepancy.

Avamar server Log Example:

On storage node 3, the log shows that the start time is a few seconds in the future:

{0.2} [srvm-17638656#srv:549]  securesessionticket::isvalid bad start time
{0.2} [srvm-17638656#srv:549]  WARN: <1410> servloop::secureticketlogin invalid sessionticket 0x45c9280 clientaddr=1.2.3.4:40755 ismaint=0 type=unknown access=<noaccess> avail=modes=0000


Background:  For VM and SQL (and other plugins), Avamar's Avagent service spawns two processes at the start of a backup/restore job.

  1. The plugin process (avvbimage or avsql)
  2. The avtar process for secure progress, which is used to obtain the work order for the plugin.
In normal circumstances, the avagent.log follows this pattern: 
avagent Info <5964>: Requesting work from <avamarserver> 
avagent Info <6686>: Process 6373 (/usr/local/avamarclient/bin/avvcbimage) for workorder MOD-1655480090808 started
avagent Info <18948>: Using secure session tickets and progress avtar for workorder processing 
avagent Info <6686>: Process 6376 (/usr/local/avamarclient/bin/avtar.bin) for workorder progress-MOD-1655480090808 started
The Avamar UI does not fetch the avtar secure progress log. For VM backups, this file is located in the /usr/local/avamarclient/var/clientlogs folder, with a log filename such as: 
/usr/local/avamarclient/var/clientlogs/progress-<wid>-<cid>-1001-Unix.log
If the avtar progress process immediately fails, it does not connect back to the plugin CTL ports, leaving the plugin in a stopped responding state. 
avtar Info <5581>: Logging in on connection 0 with Session Ticket
avtar Info <18854>: Using Secure Session Ticket Format
avtar Error <8609>: Connection killed from GSAN.
avtar Info <9772>: Starting graceful (staged) termination, KILL event received (wrap-up stage)
avtar Error <5126>: Login error 3: MSG_ERR_INVALID_PARAMETERS (Session Ticket login)
avtar FATAL <8941>: Fatal server connection problem, aborting initialization. Verify correct server address and login credentials.
avtar Info <6149>: Error summary: 3 errors: 8941, 5126, 8609
avtar Info <8469>: Progress avtar, no wrapup.
avtar Info <5314>: Command failed (3 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure))
-------------------------------------------------------------------
-- END avtar log <date>  (0 warnings, 3 errors, 1 fatal error)
-------------------------------------------------------------------

Resolution

Correct the time difference between storage nodes and the Avamar Utility Node by running the asktime procedure as detailed in KB article 000163671 

Additional Information

For similar issues, see KB article 000200098


Affected Products

Avamar
Article Properties
Article Number: 000201106
Article Type: Solution
Last Modified: 06 Feb 2025
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.