PowerStore: Unexpected reboot due to failure of systemd-tmpfiles-clean.service

Unexpected Node panic with no dump file created.

Journal logs leading up to the panic:

Sep 05 12:28:11.244196 [Node] ha_monitor_bsc[36240]: processing failed unit systemd-tmpfiles-clean.service
Sep 05 12:28:11.328465 [Node] ha_monitor_bsc[36245]: systemd-tmpfiles-clean.service failed, forcing start of BSCFailHandler@systemd-tmpfiles-clean.service
Sep 05 12:28:11.336691 [Node] bsc-fail_handler[36247]: detected failure of systemd-tmpfiles-clean
Sep 05 12:28:11.899441 [Node] serviceability_service[14677]: [S12Y_WT-20]:command_handler.py:83:INFO: Command run for CollectProcessUsage returned: {'status': 'OK'}
Sep 05 12:28:11.899556 [Node] serviceability_service[14677]: [S12Y_WT-20]:command_handler.py:127:INFO: Removing job: [{'thread_name': 'S12Y_WT-20', 'job_id': 'cb6b81cb-5350-47ea-93c2-98fac7fbf469', 'start_time': 1599308890.202333, 'end_time': 1599308891.899457, 'duration': 1.6971240043640137, 'command_type': u'CollectProcessUsage'}] from job list
Sep 05 12:28:12.067156 [Node] serviceability_service[14677]: [S12Y_WT-20]:command_handler.py:130:INFO: [0] Remaining Jobs in the S12Y service:
Sep 05 12:28:12.067292 [Node] serviceability_service[14677]: [S12Y_WT-20]:command_handler.py:97:INFO: In callback Response for command CollectProcessUsage = [{"status": "OK"}]
Sep 05 12:28:12.068184 [Node] schedule_subsystem[14677]: [S12Y_WT-SCHED]:scheduler_subsystem.py:77:INFO: Entering with task [CollectProcessUsage] next run [1599308890.0]
Sep 05 12:28:12.068370 [Node] schedule_subsystem[14677]: [S12Y_WT-SCHED]:scheduler_subsystem.py:100:INFO: Next task is [CollectProcessUsage] scheduled to run in [59.9999089241] seconds at [Sat, 05 Sep 2020 12:29:12 +0000]
Sep 05 12:28:12.082681 [Node] bsc-fail_handler[36247]: systemd-tmpfiles-clean.service failed with status ● systemd-tmpfiles-clean.service - Cleanup of Temporary Directories
Sep 05 12:28:12.082681 [Node] bsc-fail_handler[36247]:    Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.service; static; vendor preset: disabled)
Sep 05 12:28:12.082681 [Node] bsc-fail_handler[36247]:    Active: failed (Result: timeout) since Sat 2020-09-05 12:28:01 UTC; 9s ago
Sep 05 12:28:12.082681 [Node] bsc-fail_handler[36247]:      Docs: man:tmpfiles.d(5)
Sep 05 12:28:12.082681 [Node] bsc-fail_handler[36247]:            man:systemd-tmpfiles(8)
Sep 05 12:28:12.082681 [Node] bsc-fail_handler[36247]:  Main PID: 50995 (code=exited, status=0/SUCCESS)
Sep 05 12:28:12.271790 [Node] bsc-fail_handler[36247]: requesting to reset node due to failure of systemd-tmpfiles-clean.service
Sep 05 12:28:12.277677 [Node] ha_monitor_bsc[36291]: processed failed unit systemd-tmpfiles-clean.service
Sep 05 12:28:12.279226 [Node] policy[36288]: other_bsc_service has failed, policy: critical
Sep 05 12:28:12.284472 [Node] plat_event[36294]: {"name":"PLATFORM_BOOT_CONTAINER_FAILURE","ec":"0x00400105","id":"FTWSP202100646","meta":{"comp":"cyc_bsc_control","which":"A"}}
Sep 05 12:28:12.286217 [Node] policy[36295]: rebooting A
Sep 05 12:28:12.318878 [Node] xtremapp-pm[16909]: Sep 05 12:28:12.318816 P [log_id:42069][2571(2885 pm_ext_msgs      0x7f8707e6db00)]read_msg:98: ext msg. size=65 orch_ha request=reboot sender=other_bsc_service reason=ha_policy
Sep 05 12:28:12.319091 [Node] xtremapp-pm[16909]: Sep 05 12:28:12.319045 P [log_id:42312][2571(2629 nb_truck_0_pm    0x7f8707e6af40)]pm_ha_ext_msg_handler:205: orchestration msg=orch_ha request=reboot sender=other_bsc_service reason=ha_policy
Sep 05 12:28:12.319194 [Node] xtremapp-pm[16909]: Sep 05 12:28:12.319111 P [log_id:42327][2571(2629 nb_truck_0_pm    0x7f8707e6af40)]pm_node_set_reboot_required:5953: request info changed from: MGMT_FENCE_REQUEST_NONE to: MGMT_FENCE_REQUEST_REBOOT

The node should recover without any user intervention, no action required post reboot.

This issue is resolved in PowerStoreOS Service Pack 3 (SP3) v1.0.3.0.5.007.