Avamar : DialHome Symptom Code 15

The below event will be triggered and logged as a Dial Home event, an MC GUI event, or seen via a PuTTY session on the utility node.

Clarify Id: 1123XXXXXXXXXXXX
Site Name: Dell_Texas
Vendor: Dell
DeviceType: Avamar
SerialNumber: AMXXXXXXXX12
Platform: amd64
OS: Linux
EmbedLevel: 1
InternalMaxSize: 0
Ucode_Ver: 19.X.X-XXXX
IP_Address: Not Available
State: NA

SymptomCode: 15
Category: Status
Severity: Warning
Status: Warning
Component: DPN:Unknown
CDATA: <event category="SYSTEM" domain="/" eventCode="15" eventId="2816212" publishTime="173XXXXXX0000" severity="PROCESS" 
summary="node 0.X was online, changing" 
swsource="DPN:Unknown" type="WARNING"><event-source NodeID="Demo2-internal" ProgramName="com.avamar.mc.dpn.ErrorLogCache" gsan-version="19.X.X-XXXX" hardware-id="" source-hardware-id="FCXXXXXXXX01_X00-XX0-6XX-XX"/></event><CustomerContactInfo><GridInfo><site_id>000001X</site_id><grid_location>Round Rock TX US</grid_location><system><host_name>Demo1.dell.com/host_name><system_id>123456@0X:1X:2X:3X:4X:5X</system_id><ave>false</ave><ave_type>unknown</ave_type><datadomain_sn>AXXX123456</datadomain_sn></system><notes>$notes</notes></GridInfo><CustomerInfo><company><company_name>dell</company_name><company_address><street>Round Rock TX US </street><city>Fort Worth</city><state>TX</state><zipPostal>0012</zipPostal><country>USA</country></company_address></company><admin><admin_name>admin</admin_name><admin_number>20X-50X-30XX</admin_number><admin_email>exch-lab@dell.com</admin_email></admin></CustomerInfo></CustomerContactInfo>

The alert informs that there was a change in the status of the node. A node's status can change and become online due to various reasons.
Some of these are caused during and after the server maintenance activities such as SP replacement, node replacement, DIMM replacement, or power off/on.

Verify the status of nodes by running the below commands.

• Login to Avamar utility Node as admin user
  
  
• Load the SSH keys
  # ssh-agent bash
# ssh-add ~/.ssh/dpnid
  
  admin@demo1:~/>: ssh-agent bash
  admin@demo1:~/>: ssh-add ~/.ssh/dpnid
  Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)

• Ensure the status of the all the nodes are showing "ONLINE" and Access-status as "full" as highlighted below. 
  # status.dpn
  

  admin@demo1:~/>: status.dpn
  Tue Apr  1 04:10:40 EDT 2025  [demo1-1X3.lab.dell.com] Tue Apr  1 08:10:40 2025 UTC (Initialized Fri Aug 26 22:10:38 2022 UTC)
  Node   IP Address     Version   State   Runlevel  Srvr+Root+User Dis Suspend Load UsedMB Errlen  %Full   Percent Full and Stripe Status by Disk
  0.0  192.XXX.XXX.1 19.X.X-XXXX  ONLINE fullaccess mhpu+0hpu+0hpu   1 false   0.34 16908 25737142   0.1%   0%(onl:62 )  0%(onl:44 )  0%(onl:54 )  0%(onl:36 )  0%(onl:38 )  0%(onl:37 )  0%(onl:37 )  0%(onl:42 )  0%(onl:33 )  0%(onl:32 )  0%(onl:30 )  0%(onl:30 )
  0.1  192.XXX.XXX.2 19.X.X-XXXX  ONLINE fullaccess mhpu+0hpu+0hpu   2 false   0.28 17073 25540676   0.1%   0%(onl:40 )  0%(onl:33 )  0%(onl:41 )  0%(onl:33 )  0%(onl:68 )  0%(onl:32 )  0%(onl:33 )  0%(onl:39 )  0%(onl:31 )  0%(onl:36 )  0%(onl:31 )  0%(onl:50 )
  0.2  192.XXX.XXX.3 19.X.X-XXXX  ONLINE fullaccess mhpu+0hpu+0hpu   0 false   0.31 15332 25683208   0.1%   0%(onl:36 )  0%(onl:39 )  0%(onl:41 )  0%(onl:48 )  0%(onl:40 )  0%(onl:34 )  0%(onl:41 )  0%(onl:32 )  0%(onl:41 )  0%(onl:35 )  0%(onl:34 )  0%(onl:46 )
  Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

  System ID: 123456@0X:1X:2X:3X:4X:5X

  All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
  System-Status: ok
  Access-Status: full

  Last checkpoint: cp.2025XXXXXXXXXX1 finished Mon Mar 31 12:02:55 2025 after 00m 25s (OK)
  No GC yet
  Last hfscheck: finished Mon Mar 31 12:02:19 2025 after 02m 52s >> checked 1406 of 1406 stripes (OK)

  Maintenance windows scheduler capacity profile is active.
    The backup window is currently running.
    Next backup window start time: Tue Apr  1 20:00:00 2025 EDT
    Next maintenance window start time: Tue Apr  1 08:00:00 2025 EDT

• Make sure the node's are showing uptime
  # mapall --all+ --noerror uptime
  
  admin@demo1:~/>: mapall --all+ --noerror uptime
  Using /usr/local/avamar/var/probe.xml
  (0.s) ssh -q  -x  -o GSSAPIAuthentication=no admin@192.XXX.XXX.0 'uptime'
   04:10am  up  17:43,  1 user,  load average: 0.26, 0.33, 0.30
  (0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@192.XXX.XXX.1 'uptime'
   08:10am  up  18:11,  0 users,  load average: 0.49, 0.25, 0.21
  (0.1) ssh -q  -x  -o GSSAPIAuthentication=no admin@192.XXX.XXX.2 'uptime'
   08:10am  up  18:11,  0 users,  load average: 0.37, 0.21, 0.19
  (0.2) ssh -q  -x  -o GSSAPIAuthentication=no admin@192.XXX.XXX.3 'uptime'
   08:10am  up  18:11,  0 users,  load average: 0.39, 0.22, 0.19

• Verify the services are up and running fine
  # dpnctl status
  
  admin@demo1:~/>: dpnctl status
  dpnctl: INFO: gsan status: up
  dpnctl: INFO: MCS status: up.
  dpnctl: INFO: emt status: up.
  dpnctl: INFO: Backup scheduler status: up.
  dpnctl: INFO: Maintenance windows scheduler status: enabled.
  dpnctl: INFO: avinstaller status: up.
  dpnctl: INFO: ddrmaint-service status: up.
  dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
  

• Ensure we have the latest 2 Valid checkpoint with 1 being rol/hfs checkpoint
  # cplist --lscp
  
  admin@demo1:~/>: cplist --lscp
  cp.2025XXXXXXXXXX0 Mon Mar 31 11:58:40 2025   valid rol ---  nodes   3/3 stripes   1409
  cp.2025XXXXXXXXXX1 Mon Mar 31 12:02:30 2025   valid --- ---  nodes   3/3 stripes   1409

• If the above output matches and node health is online, acknowledge the alerts from the GUI/AUI
  
  

If the node health is offline/down, generate a support bundle, attach it to the SR and request for assistance on the Alert Response Form.
KB to generate Support Bundle : Avamar: How to run "getlogs" to gather Avamar server logs.