Data Domain: DDVE reboots automatically every hour

messages.engineering
Apr 15 10:23:42 localhost syslog: NOTICE: : DDOS 6.1.2.10-603959 booting up
Apr 15 11:20:32 localhost syslog: NOTICE: : DDOS 6.1.2.10-603959 booting up
Apr 15 12:21:53 localhost syslog: NOTICE: : DDOS 6.1.2.10-603959 booting up
Apr 15 13:27:48 localhost syslog: NOTICE: : DDOS 6.1.2.10-603959 booting up
Apr 15 14:33:49 localhost syslog: NOTICE: : DDOS 6.1.2.10-603959 booting up
... 
Apr 17 07:29:30 localhost syslog: NOTICE: : DDOS 6.1.2.10-603959 booting up
 

messages.engineering
Apr 15 10:25:12 ddve sms: {epoch=1555313112;id='AUDIT-SMS_REPLICATION-00050';desc='Reset replication throttle schedule.';level=1;user='(null)';role='unknown';app='bootbin';host='127.0.0.1';detail='{err_msg='User not authorized to execute this command: The session ID is missing!';err_num=5320;in='{type=1;group=-1;}';}';}
Apr 15 10:25:15 ddve sms: {epoch=1555313115;id='AUDIT-SMS_NET-00013';desc='Get DNS Servers.';level=1;user='(null)';role='unknown';app='netmon';host='127.0.0.1';detail='{err_msg='User not authorized to execute this command: The session ID is missing!';err_num=5320;}';}
...
Apr 17 07:30:45 ddve sms: {epoch=1555475445;id='AUDIT-SMS_REPLICATION-00050';desc='Reset replication throttle schedule.';level=1;user='(null)';role='unknown';app='bootbin';host='127.0.0.1';detail='{err_msg='User not authorized to execute this command: The session ID is missing!';err_num=5320;in='{type=1;group=-1;}';}';}







 

The Following Symptoms have been noted to cause this issue:

1. The DDVE was taken down for a period of time

Apr 15 09:32:57 ddve -ddsh: NOTICE: MSG-DDSH-00009: (tty=tty1, session=20725) sysadmin: command "system poweroff"
...
Apr 15 10:23:42 localhost syslog: NOTICE: : DDOS 6.1.2.10-603959 booting up


2. The VMWare virtual hardware environment was upgraded from version 10 to version 11
- Confirm this with the customer

Version 10                   Version 11
ESXi 5.5                      ESXi 6.0
Workstation 10.x         Workstation 11.x


3. On the subsequent reboot, the System hits Bug#223300 
- Note: This issue is fixed in DD OS 6.1.2.20
Utilization.report uses logger to send the hourly status to syslogd.
Syslogd will write the status to both the log file and the console. 
But somehow syslogd failed to write the message to console which makes syslogd enter infinite loop. 
Later on when crond runs cron job a log message will be sent to syslogd as well. 
But as syslogd is stuck in the infinite loop that operation never finishes.
This makes crond stuck as well. 
So the kernel watchdog cannot be touched. 
After 900 seconds kernel panics, resulting in reboot of DDVE.

Tech Support can provide a workaround for this; please open a support case with Data Domain.
The workaround includes a system reboot after the required fix has been applied.
Alternatively, consider upgrading to DD OS 6.1.2.20