ECS: xDoctor RAP137 Total swap memory inconsistent across the ECS rack

xDoctor reports:

------------------------------------------------------------
WARNING - Total swap memory inconsistent across the ECS rack
------------------------------------------------------------
Node      = Nodes
Extra     = {'Nodes': {'16383': ['169.254.1.8', '169.254.1.7', '169.254.1.6', '169.254.1.4', '169.254.1.3', '169.254.1.2', '169.254.1.1'], '8191': ['169.254.1.5']}}
Timestamp = 2022-12-01_145111
PSNT      = CKM00190800120 @ 4.8-87.0

• After a node reboot, the SWAP configuration was not saved in /etc/fstab.
• The native SWAP partition did not loaded at boot.
• The node was reimaged, and SWAP was not applied to the node.
• More SWAP was never applied to a cluster that had an expansion. 

1. Confirm if SWAP consistency on all the nodes.

Command:

# doit free -m | grep Swap

Example:

admin@ecsnode1:~> svc_exec 'free -m | grep Swap'
svc_exec v1.0.4 (svc_tools v2.8.0)                 Started 2022-11-30 22:15:56

Output from node: r1n1                                retval: 0
Swap:        8191         552         23

Output from node: r1n2                                retval: 0
Swap:        8191          2        8191

Output from node: r1n3                                retval: 0
Swap:        16383          4      16379

Output from node: r1n4                                retval: 0
Swap:        16383          0      16383

Output from node: r1n5                                retval: 0
Swap:        16383          0      16383

Output from node: r1n6                                retval: 0
Swap:        16383          3      16380

Output from node: r1n7                                retval: 0
Swap:        16383          4      16379

Output from node: r1n8                                retval: 0
Swap:        16383        699      15684

2. Confirm if /etc/fstab is updated to use additional 8 GB of SWAP on the node and if the native SWAP partition entry exists.

Command:

# svc_exec 'grep swap /etc/fstab'

Example:

admin@ecsnode1:~> svc_exec 'grep swap /etc/fstab'
svc_exec v1.0.4 (svc_tools v2.8.0)                 Started 2022-11-30 22:19:21

Output from node: r1n1                                retval: 0
LABEL=SWAP swap swap defaults 0 0                     <-- missing extended swap partition

Output from node: r1n2                                retval: 0
/swapfile     swap swap defaults 0 0                  <-- missing native swap partition

Output from node: r1n3                                retval: 0
LABEL=SWAP swap swap defaults 0 0
/swapfile     swap swap defaults 0 0

Output from node: r1n4                                retval: 0
LABEL=SWAP swap swap defaults 0 0
/swapfile     swap swap defaults 0 0

Output from node: r1n5                                retval: 0
LABEL=SWAP swap swap defaults 0 0
/swapfile     swap swap defaults 0 0

Output from node: r1n6                                retval: 0
LABEL=SWAP swap swap defaults 0 0
/swapfile     swap swap defaults 0 0

Output from node: r1n7                                retval: 0
LABEL=SWAP swap swap defaults 0 0
/swapfile     swap swap defaults 0 0

Output from node: r1n8                                retval: 0
LABEL=SWAP swap swap defaults 0 0
/swapfile     swap swap defaults 0 0

3. Validate if the SWAP file is present on all nodes.

# svc_exec 'ls -lh /swapfile'

admin@ecsnode1:~> svc_exec 'ls -lh /swapfile'
svc_exec v1.0.4 (svc_tools v2.8.0)                 Started 2022-11-30 22:26:48

Output from node: r1n1                                retval: 0
ls: cannot access '/swapfile': No such file or directory

Output from node: r1n2                                retval: 0
-rw------- 1 root root 8.0G Feb 11  2022 /swapfile

Output from node: r1n3                                retval: 0
-rw------- 1 root root 8.0G Feb 11  2022 /swapfile

Output from node: r1n4                                retval: 0
-rw------- 1 root root 8.0G Feb 11  2022 /swapfile

Output from node: r1n5                                retval: 0
-rw------- 1 root root 8.0G Feb 11  2022 /swapfile

Output from node: r1n6                                retval: 0
-rw------- 1 root root 8.0G Feb 11  2022 /swapfile

Output from node: r1n7                                retval: 0
-rw------- 1 root root 8.0G Feb 11  2022 /swapfile

Output from node: r1n8                                retval: 0
-rw------- 1 root root 8.0G Jul 26 02:03 /swapfile

IMPORTANT! A new feature has been released in xDoctor 4-8.104.0 and above. This knowledge base (KB) is now automated with xDoctor AutoPilot addressing most issues without the need for support involvement.

This feature is native to xDoctor 4-8.104.0 and above, for syntax and usage issues reference ECS: ObjectScale: How to run KB Automation Scripts (Auto Pilot).(may require login to Dell Support) 
 

1. Run the automation command from a primary node with xDoctor 4-8.104.0 and above.
   Note: --target-node, --target-rack and --target-vdc is supported for this script.

# sudo xdoctor autopilot --kb 205933 --target-node <NAN IP>
# sudo xdoctor autopilot --kb 205933 --target-rack <Rack Name>
# sudo xdoctor autopilot --kb 205933 --target-vdc <VDC Name>

admin@ecsnode1:~> sudo xdoctor autopilot --kb 205933 --target-node 169.254.1.1
Checking for existing screen sessions...
Starting screen session 'autopilot_kb_205933_20250627_212503'...
Screen session 'autopilot_kb_205933_20250627_212503' started successfully.
Attaching to screen session 'autopilot_kb_205933_20250627_212503'...

2. Review a summary of actions taken.

Example:

TASK [Summary for SWAP and fstab issues addressed] *********************************************************************************************************************************************
ok: [169.254.1.7] => {
    "msg": [
        "************************************************************************************************************************************************",
        "************************************************************************************************************************************************",
        " An 8GB swap file was created and configured as /swapfile with swap enabled. ",
        " fstab was updated with entries for /swapfile, and a backup prior to the change was created in: /etc/fstab.28734.2024-08-28@19:44:38~. ",
        " No duplicate entries were found in fstab. ",
        "************************************************************************************************************************************************",
        "************************************************************************************************************************************************"
    ]
}

4. Confirm to update the MOTD with swap details and the changes applied.
   Note: If your current MOTD has the KB number or the word /swapfile in it the MOTD will not be updated.

Command:

# yes or no

Example:

The next steps are to detect keywords in the MOTD content and update the MOTD with custom content for the KB specifics. IMPORTANT! You will need customer permission to do this action for the KB. Would you like to proceed with the steps by typing 'Yes' or 'Y', or skip the MOTD update actions by typing 'No' or 'N':
: y

5. Review MOTD change summary for adding new MOTD content:

Example:

TASK [MOTD summary] ****************************************************************************************************************************************************************************
ok: [169.254.1.7] => {
    "msg": [
        "************************************************************************************************************************************************",
        "New MOTD content:",
        "************************************************************************************************************************************************",
        "************************************************************************************************************************************************",
        [
            "",
"==============================================================================================================================================",
            "2024-08-28 | SR#84955203 | KB 186358 has been applied to this system, adding an 8GB swap file to /swapfile. Please reapply if a node has been reimaged.",
            "=============================================================================================================================================="
        ],
        "************************************************************************************************************************************************",
        "************************************************************************************************************************************************"
    ]
}