CloudLink: A CloudLink node is stuck in emergency mode

After booting the CloudLink node, the following message is seen: 

This issue is due to a file system issue that must be repaired.

To resolve the file system issue on the node, perform the following:

1. You press Control-D to continue, but it remains in emergency mode.
2. Press the ENTER key for maintenance.
3. Run a file system check on the root (  /  ) file system.

root@cloudlink:~#  lsblk    <------------------- Run this command to identify the / filesystem (typically sda10)
root@cloudlink:~#  fsck /dev/sda10  <-------- It complains that /dev/sda10 is still mounted
root@cloudlink:~#  umount /dev/sda10
root@cloudlink:~#  fsck /dev/sda10

4. Press y to fix any errors that are found.
5. Reboot the node.

If the root filesystem is clean, you can run fsck on the other filesystems using the same procedure above.
Run command journalctl -xb. These logs might indicate if there is a corrupted filesystem that needs repaired.
You can use commands lsblk and blkid to help link a uuid to its filesystem.
Here are some examples of errors that you might see in journalctl -xb:
/dev/sda6: Inodes that were part of a corrupted orphan linked list found.
/dev/sda6: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY.
Failed to start File System Check on /dev/disk/by-uuid/fd782a8-a70d-045513691c7e

If you are unable to repair the filesystem using fsck or if you have one Cloudlink node that is frequently getting stuck in emergency mode, you should remove that broken Cloudlink node from the cluster. Then you deploy a new Cloudlink node from OVA and join to the existing cluster.

If you can repair the file system, you should check the health of the cluster from the Cloudlink UI.
This is done in SYSTEM > Cluster and ensure incoming, and outgoing sync states are OK.