Dell EMC Unity: SP rolling reboot due to eVE failure (EMC Correctable)
This article applies to
This article does not apply to
This article is not tied to any specific product.
Not all product versions are identified in this article.
Symptoms
Someone created a file "ls" in /var/lib/docker/devicemapper/metadata by mistake.
SP rolling reboot with eVE errors in /EMC/C4Core/log/start_c4.log (/mnt/ssdroot/EMC/C4Core/log/start_c4.log if SP is in service mode):
Mon Aug 7 21:51:41 UTC 2017 CF: iwd: fully started in 0 seconds
Mon Aug 7 21:54:05 UTC 2017 CF: HA: eVE: Error: job cf.eVE stopped
Mon Aug 7 21:54:05 UTC 2017 CF: HA: eVE: stopping
Mon Aug 7 21:54:06 UTC 2017 CF: HA: eVE: Error: failed to stop eVE
Mon Aug 7 21:54:11 UTC 2017 c4_shutdown.sh: called with "-f"
SP rolling reboot with eVE errors in /EMC/C4Core/log/start_c4.log (/mnt/ssdroot/EMC/C4Core/log/start_c4.log if SP is in service mode):
Mon Aug 7 21:51:41 UTC 2017 CF: iwd: fully started in 0 seconds
Mon Aug 7 21:54:05 UTC 2017 CF: HA: eVE: Error: job cf.eVE stopped
Mon Aug 7 21:54:05 UTC 2017 CF: HA: eVE: stopping
Mon Aug 7 21:54:06 UTC 2017 CF: HA: eVE: Error: failed to stop eVE
Mon Aug 7 21:54:11 UTC 2017 c4_shutdown.sh: called with "-f"
Cause
According to /EMC/CEM/log/eVE/eVE_startup.log on SP, docker is not running.
Mon Aug 7 21:49:05 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not running
Mon Aug 7 21:49:05 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not running
Mon Aug 7 21:49:06 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not running
According to /var/log/messages, docker service failed because it could not load device file "ls".
2017-08-07T21:41:09+00:00 self docker[18109]: time="2017-08-07T21:41:09.996387324Z" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devmapper: Failed to load device files:Error looking up device ls:Unknown device ls"
2017-08-07T21:41:09+00:00 self docker[18109]: time="2017-08-07T21:41:09.996422502Z" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Failed to load device files:Error looking up device ls:Unknown device ls"
2017-08-07T21:41:10+00:00 self systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mon Aug 7 21:49:05 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not running
Mon Aug 7 21:49:05 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not running
Mon Aug 7 21:49:06 2017 /EMC/Platform/bin/eVE.pl: Cannot proceed. Docker service is not running
According to /var/log/messages, docker service failed because it could not load device file "ls".
2017-08-07T21:41:09+00:00 self docker[18109]: time="2017-08-07T21:41:09.996387324Z" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devmapper: Failed to load device files:Error looking up device ls:Unknown device ls"
2017-08-07T21:41:09+00:00 self docker[18109]: time="2017-08-07T21:41:09.996422502Z" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Failed to load device files:Error looking up device ls:Unknown device ls"
2017-08-07T21:41:10+00:00 self systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Resolution
1. Remove the problematic file in /var/lib/docker/devicemapper/metadata.
rm -f /var/lib/docker/devicemapper/metadata/ls
2. Clear the boot rescue counter.
svc_rescue_state -c
3. Reboot the SP.
svc_shutdown -r
Note: SP re-image can also resolve this problem.
rm -f /var/lib/docker/devicemapper/metadata/ls
2. Clear the boot rescue counter.
svc_rescue_state -c
3. Reboot the SP.
svc_shutdown -r
Note: SP re-image can also resolve this problem.
Affected Products
Dell EMC Unity FamilyProducts
Dell EMC Unity FamilyArticle Properties
Article Number: 000050042
Article Type: Solution
Last Modified: 10 Oct 2024
Version: 3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.