Avamar: hfscheck fail with error MSG_ERR_CMD_FAIL

Avamar maintenance activity shows failed hfscheck, preventing the checkpoint to be validated. 
 

# status.dpn

Last checkpoint: cp.20260121214733 finished Wed Jan 21 16:47:51 2026 after 00m 18s (OK)
Last GC: finished Wed Jan 21 08:00:49 2026 after 00m 05s >> recovered 21.08 KB (OK)
Last hfscheck failed with result MSG_ERR_CMD_FAIL : started Wed Jan 21 16:47:59 2026 ended Wed Jan 21 16:48:07 2026, nodes queried/replied/total(1/1/1)

Review of gsan logs display error presented. 
 

/data01/cur/gsan.log
2026/01/21-21:48:07.92035 {0.0} [cprecovery:133]  checkpoint::dnscontext::resetdns using alternate stripedns srcdir=recover_do_not_touch/cp.20260121214733.1769032053.20260121T214807x920059 dns=cur
2026/01/21-21:48:07.92038 {0.0} [manage:1279]  negotiator::request exception server_exception(MSG_ERR_CMD_FAIL)
2026/01/21-21:48:07.92038 {0.0} [manage:1279]  dpncheckmanager::dpncheckcontext::revert
2026/01/21-21:48:07.92041 {0.0} [cprecovery:133]  stripetab::stripetab sizeof(stripelistitem) = 160
2026/01/21-21:48:07.92042 {0.0} [cprecovery:133]  stripetab::stripetab sizeof(stripeitemtype) = 168
2026/01/21-21:48:07.92042 {0.0} [cprecovery:133]  nodetab::nodetab sizeof(nodelistitem) = 496
2026/01/21-21:48:07.92043 {0.0} [cprecovery:133]  nodetab::nodetab sizeof(nodeitemtype) = 512
2026/01/21-21:48:07.92043 {0.0} [manage:1279]  samthread::call server_exception server_exception(MSG_ERR_CMD_FAIL) REQUESTDP

Observation of "home/admin" directory permission is applicable in this case. 

root@hop-ave-81-v1912:/home/#: ls -la
total 4
drwxr-xr-x  7 root   root     69 Mar  3  2025 .
drwxr-xr-x  4 root   root     32 Mar  3  2025 ..
drwxr-x--- 17 root   admin  4096 Jan 21 09:01 admin
drwxr-xr-x  8 avi    avi     204 Dec 17  2024 avi
drwxr-x---  3 dpn    admin   119 Apr 11  2025 dpn
drwxr-xr-x  8 tomcat tomcat  198 Sep 26 13:52 tomcat
drwxr-xr-x  6 vauser vauser  141 Dec 17  2024 vauser

• Strict ownership checks
  Avamar’s health checks and checkpoint logic validate the runtime environment, including:

  • Ownership of /home/admin and all contents must be admin:admin.
  • The SSH materials (for example, /home/admin/.ssh/authorized_keys, keys) must be owned by admin and have strict perms (no write for group/others).
  • Some scripts and runtime state written under /home/admin must be accessible by the admin user and not owned by root.

  Changing root:admin, dpncheck/cprecovery detect unexpected ownership and bail out as a safety measure.

• Negotiator/management dependency on admin’s home
  The Avamar negotiator and management daemons rely on the admin account context for auth and orchestration. If files under /home/admin are not owned by admin, you can see:

  • SSH auth failures or skipped execution due to insecure perms.
  • Scripts refusing to run when they detect the wrong owner (defense-in-depth).
  • Resulting command failures bubble up as MSG_ERR_CMD_FAIL.

1. Login as root to change /home/admin permissions. 

# su -
# chmod root:admin /home/admin

root@hop-ave-81-v1912:/home/#: ls -la
total 4
drwxr-xr-x  7 root   root     69 Mar  3  2025 .
drwxr-xr-x  4 root   root     32 Mar  3  2025 ..
drwxr-x--- 17 admin  admin  4096 Jan 21 09:01 admin
drwxr-xr-x  8 avi    avi     204 Dec 17  2024 avi
drwxr-x---  3 dpn    admin   119 Apr 11  2025 dpn
drwxr-xr-x  8 tomcat tomcat  198 Sep 26 13:52 tomcat
drwxr-xr-x  6 vauser vauser  141 Dec 17  2024 vauser