PowerPath for Linux: PowerPath does not start after reboot and must be manually started

Operating System: Red Hat Enterprise Linux 7.x, OEL 7.x, and SLES 12
This does not apply for versions of Red Hat Enterprise Linux 5.x, 6.x and SLES 11.
EMC Software: PowerPath for Linux 6.x

PowerPath does not start automatically after the reboot of the host.
PowerPath can be started manually without any issue.
No errors are logged in boot.log when the PowerPath service starts and stops.

After the system reboots, you would see that the PowerPath fails with "Initialization Error" since the service was unable to start:

# powermt display dev=all ERROR: Cannot open PowerPath. Initialization error
# cat /var/log/boot.log |grep Power Starting PowerPath startup script... [ OK ] Stopped PowerPath startup script. <--- Starting PowerPath Management Daemon... Starting PowerPathPost startup script... [ OK ] Started PowerPathPost startup script. [ OK ] Started PowerPath Management Daemon.

On start, PowerPath has a few checks initially before it proceeds with actually starting. One of these is to check when PowerPath was last started and this is determined by calculating the age of a marker file. On bootup, the system clock is initially synced from the hardware clock. However, if an NTP client is configured on the host, then when that starts, the system clock is updated from an NTP server.

The problem arises if the hardware clock is too much out of sync with that of the NTP clock, usually when it is much behind. On host bootup, PowerPath starts much before the NTP client. Hence, when it tries to determine when it tries to calculate the age of the marker file, it is a negative value and PowerPath exits immediately without proceeding with the start.

Workaround:
Update the hwclock to the system clock using the following command on the host:

hwclock --systohc

This updates the hardware clock to the correct time on the host.

Permanent Fix:
The issue will be fixed in PowerPath for Linux 6.3.

You can run the below command to check the time when PowerPath was last started:

systemctl status PowerPath.service

Usually you would see that the last time when PowerPath was active has a huge time difference:

systemctl status PowerPath.service   PowerPath.service - PowerPath startup script Loaded: loaded (/usr/lib/systemd/system/PowerPath.service; enabled; vendor preset: disabled) Active: inactive (dead) since Thu 2017-04-06 12:00:15 GET; 4h 0min ago Process: 2075 ExecStopPost=/usr/lib/PowerPath/PowerPath check_service_stop (code=exited, status=0/SUCCESS) Process: 1807 ExecStartPost=/usr/lib/PowerPath/PowerPath check_service_start (code=killed, signal=TERM) Process: 1665 ExecStart=/usr/lib/PowerPath/PowerPath start (code=exited, status=0/SUCCESS) Main PID: 1665 (code=exited, status=0/SUCCESS)

------------------------------------------------------------------------------------------
Another way to find is the enable '-x' flag in the PowerPath startup script.
For OEL 7.x, the startup scripts are:

/usr/lib/PowerPath/PowerPath

/etc/opt/emcpower/EMCpower.LINUX-6.1.0.00.00/enable

Example:

# head -5 /usr/lib/PowerPath/PowerPath
#!/bin/sh -x

Once you reboot the host, the logs would be captured in boot.log. Also run the below command where last_boot would have the extract of the same:

journalctl -b 0 > last_boot

In the below verbose logs which were collected above, we see the time lag of -13991.

Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + '[' yes == yes -a -f /tmp/.pp_systemctl_check ']'
Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: ++ date +%s
Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: ++ date +%s -r /tmp/.pp_systemctl_check
Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + '[' -13991 -lt 5 ']'
Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + rm -f /tmp/.pp_systemctl_check
Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + /bin/true
Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + rc_exit
Apr 20 10:23:23 revdb1.mfg.ge PowerPath[1751]: + exit 0