Dell NativeEdge: Manual Witness Node Installation

Purpose

The Witness Node serves as a critical component in the NativeEdge distributed architecture, requiring specific network, security, and time synchronization configurations. After cluster creation, the cluster primary nodes handle the witness node's Lifecycle Management (LCM). The steps described in this document are only required to create a ready-to-be-used Witness node for NativeEdge 2-Nodes Clusters.

Prerequisites and requirements

Hardware requirements

• Processor: 64-bit x86 processor with minimum two cores
• Memory: 4-GB RAM minimum
• Storage: 20-GB available disk space minimum
• Network: Active network interface card

Software requirements

• Ubuntu 24.04 LTS Server ISO image
• Network configuration details (IP address, netmask, gateway, DNS servers)
• Customer-defined password for witness-user account (minimum eight characters and must contain letters and numbers)

Access requirements

• Physical or remote console access to the target system

Install Ubuntu 24.04 LTS Server.

1. Boot from Ubuntu 24.04 LTS ISO

1. 1. Insert the Ubuntu 24.04 LTS Server ISO media or mount it to your virtual machine
   2. Boot the system from the ISO media
   3. At the boot menu, select "Install Ubuntu Server"

2. Select language and keyboard

1. 1. Select your preferred language (default: English)
   2. Select your keyboard layout (default: English (US))
   3. Press Enter to continue

3. Configure the network

1. 1. When prompted for network configuration, select "Edit IPv4" on your primary network interface
   2. Choose "Manual" configuration method
   3. Enter the following network details:
      • Subnet: Enter your subnet in CIDR notation (for example, 192.168.1.0/24)
      • Address: Enter the static IP address for this witness node
      • Gateway: Enter the gateway IP address
      • Name servers: Enter DNS server addresses (comma-separated)
   4. Select "Save" to apply the configuration

4. Configure storage
   1. Select "Use an entire disk" for storage configuration
   2. Choose the target disk for installation
   3. Review the storage layout and select "Done"
   4. Confirm the destructive action by selecting "Continue"

5. Profile Setup

1. 1. Enter the following information:
      • Your name: Witness Administrator
      • Your server's name: witness-node (or as per your naming convention)
      • Pick a username: Installer (temporary account)
      • Choose a password: Enter a secure temporary password
      • Confirm your password: Reenter the password

6. Configure SSH

1. 1. When prompted for SSH setup, select "Install OpenSSH server"
   2. Press Space to enable the checkbox
   3. Do not import SSH identity at this stage
   4. Select "Done" to continue

7. Featured Server Snaps

1. 1. Do not select any additional snaps to maintain a minimal installation
   2. Select "Done" to finish the installation

8. Complete installation

1. 1. The installation process begins. This may take 10-15 minutes.
   2. Once complete, select "Reboot Now"
   3. Remove the installation media when prompted
   4. The system reboots into the newly installed Ubuntu 24.04 LTS Server

Post-Installation Configuration

Log in to the system using the temporary installer account created during installation. Perform the following configuration steps in order:

1. Update system packages

sudo apt update

sudo apt upgrade -y

2. Configure a static network (If not done during installation)

If network configuration was not completed during installation, configure it now:

1. 1. Identify your network interface:

ip link show

1. 2. Edit the netplan configuration file:

sudo nano /etc/netplan/00-installer-config.yaml

1. 3. Configure the static IP address (replace values with your network details):

network:
  version: 2
  ethernets:
    ens160:  # Replace with your interface name
      dhcp4: no
      addresses:
        - 192.168.1.100/24  # Replace with your IP/netmask
      gateway4: 192.168.1.1  # Replace with your gateway
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

2. 4. Apply the network configuration:

sudo netplan apply

1. 5. Verify the network configuration:

ip addr show
ip route show
ping -c 4 <gateway-ip>

3. Install the required packages

1. 1. Install all necessary packages for the witness node:

sudo apt-get install -y chrony ceph-mon ceph-common wipe iproute2 net-tools iputils-ping iptables ebtables tcpdump traceroute conntrack

1. 2. Verify the installation:

dpkg -l | grep -E 'chrony|ceph-mon|ceph-common'

4. Configure chrony time synchronization

1. 1. Configure chrony with your Network Time Protocol servers:

sudo nano /etc/chrony/chrony.conf

1. 2. Add or modify the Network Time Protocol server entries (replace with your organization's Network Time Protocol servers). At least one Network Time Protocol server (with a line starting with server) is mandatory or the witness is rejected at cluster creation:

# Dell Network Time Protocol Servers
server ntp1.dell.com iburst
server ntp2.dell.com iburst

# Fallback public Network Time Protocol servers
pool 2.ubuntu.pool.ntp.org iburst maxsources 4

# Allow local time to be used if Network Time Protocol servers are unreachable
local stratum 10

# Record the rate at which the system clock gains or loses time
driftfile /var/lib/chrony/drift

# Enable kernel synchronization of the real-time clock
rtcsync

# Step the system clock instead of slew if adjustment is larger than 1 second
makestep 1 3

1. 3. Restart and enable chrony:

sudo systemctl restart chrony
sudo systemctl enable chrony

1. 4. Verify chrony synchronization:

sudo chronyc tracking
sudo chronyc sources

5. Create witness user account

Important: This account is temporary, and NativeEdge physical nodes only use it once during cluster creation initiation. After the cluster is successfully created, the NativeEdge system automatically deletes this account.

1. 1. Create the witness-user account:

sudo adduser --disabled-password --gecos "Witness Service Account" witness-user

1. 2. Set a custom password for the witness-user account:

Note: You must provide this password during the NativeEdge cluster creation process. Store it securely until cluster initialization is complete.

• • • Minimum length: Eight characters
    • Must contain both letters (a-z, A-Z) and numbers (0-9)
    • Example: Witness2024, Node1234, Edge5678

sudo passwd witness-user

1. 3. Add the user to the sudo group and configure passwordless sudo access:

sudo usermod -aG sudo witness-user

1. 4. Configure passwordless sudo for witness-user:

echo "witness-user ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/witness-user
sudo chmod 440 /etc/sudoers.d/witness-user

1. 5. Generate SSH key pair for witness-user:

sudo -u witness-user ssh-keygen -t rsa -b 4096 -f /home/witness-user/.ssh/id_rsa -N "" -C "witness-user@$(hostname)"

1. 6. Verify SSH key generation:

sudo ls -la /home/witness-user/.ssh/

Expected output:

• • • id_rsa (private key)
    • id_rsa.pub (public key)
    • authorized_keys (for incoming connections)

6. Prepare iptables

1. 1. Add the below content in the /etc/iptables.rules with owner being root:root and permissions at 0644:

*filter
:INPUT DROP [120:8633]
:FORWARD DROP [0:0]
:OUTPUT DROP [2533:382155]

-N LOGGING_INPUT
-N LOGGING_OUTPUT
-N LOGGING_FORWARD
-N CUSTOM_FILTER_INPUT_CHAIN
-N CUSTOM_FILTER_OUTPUT_CHAIN
-N CUSTOM_FILTER_FORWARD_CHAIN

-A INPUT -i lo -j ACCEPT
-A INPUT -s 127.0.0.0/8 -j DROP
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p udp -m multiport --dports 67,68,123,53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m conntrack --ctstate INVALID -j DROP

-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m conntrack --ctstate INVALID -j DROP
-A OUTPUT -p udp -m multiport --dports 67,68,123,53 -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 53 -j ACCEPT

-A INPUT -j CUSTOM_FILTER_INPUT_CHAIN
-A INPUT -j LOGGING_INPUT

-A OUTPUT -j CUSTOM_FILTER_OUTPUT_CHAIN
-A OUTPUT -j LOGGING_OUTPUT

-A FORWARD -j CUSTOM_FILTER_FORWARD_CHAIN
-A FORWARD -j LOGGING_FORWARD

-A LOGGING_INPUT -m limit --limit 10/min -j LOG --log-prefix "IPTables-INPUT-Dropped: " --log-level 4
-A LOGGING_OUTPUT -m limit --limit 10/min -j LOG --log-prefix "IPTables-OUTPUT-Dropped: " --log-level 4
-A LOGGING_FORWARD -m limit --limit 10/min -j LOG --log-prefix "IPTables-FORWARD-Dropped: " --log-level 4
-A LOGGING_INPUT -j DROP
-A LOGGING_OUTPUT -j DROP
-A LOGGING_FORWARD -j DROP
COMMIT

1. 2. Add the below content in the file /etc/systemd/system/iptables-restore.service:

[Unit]
Description=Restore iptables rules
Before=network-pre.target
Wants=network-pre.target

[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore /etc/iptables.rules
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

1. 3. Ensure the file /etc/systemd/system/iptables-restore.service ownership is root:root and file permission being 0644
   4. Run the below command once all the values are set, to make them persistent across reboots.

      systemctl start iptables-restore
      systemctl enable iptables-restore

7. Disable Ubuntu disruptive packages and automated updates

1. 1. Run:

apt-get remove -y unattended-upgrades update-notifier-common apport snapd packagekit fwupd ubuntu-pro-client ubuntu-pro-client-l10n
systemctl disable --now apt-daily-upgrade.timer apt-daily.timer dpkg-db-backup.timer fwupd-refresh.timer motd-news.timer
systemctl disable --now ModemManager.service packagekit.service

1. 2. Reboot is recommended to ensure that all services are in place and running

Verification

Perform the following verification steps to ensure proper configuration:

1. Network Verification
   1. Verify IP address configuration

ip addr show

1. 2. Verify gateway configuration

ip route show

1. 3. Test network connectivity

ping -c 4 <gateway-ip>
ping -c 4 8.8.8.8

Expected Result: IP address, netmask, and gateway are correctly configured; ping tests succeed.

2. chrony Verification
   1. Check chrony service status

sudo systemctl status chrony

1. 2. Verify time synchronization

sudo chronyc tracking

1. 3. Check NTP sources

sudo chronyc sources -v

Expected Result: chrony service is active and running; system time is synchronized with NTP servers.

3. User Account Verification
   1. Verify witness-user exists

id witness-user

1. 2. Verify user home directory and SSH keys

ls -la /home/witness-user/

1. 3. Verify passwordless sudo configuration

sudo cat /etc/sudoers.d/witness-user

1. 4. Test user login (from another terminal)

su - witness-user

1. 5. Test passwordless sudo (as witness-user)

sudo whoami

Expected Result: User witness-user exists with proper home directory, SSH keys, and can perform sudo commands without password prompts.

4. SSH Server Verification
   1. Verify SSH service status

sudo systemctl status sshd

1. 2. Verify that SSH is listening on port 22

sudo ss -tlnp | grep :22

1. 3. Test SSH connection from remote host

ssh witness-user@<witness-node-ip>

Expected Result: SSH service is active and listening; remote SSH connection succeeds.

Optional: Install packages offline. 

If the witness server does not have Internet access, download the required packages on an Internet-connected Ubuntu system and transfer them to the witness server:

1. Download packages on an Internet-connected system:

apt-get download chrony ceph-mon ceph-common wipe iproute2 net-tools iputils-ping iptables ebtables tcpdump traceroute conntrack

2. Download dependencies:

apt-get download $(apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts --no-breaks --no-replaces --no-enhances chrony ceph-mon ceph-common wipe iproute2 net-tools iputils-ping iptables ebtables tcpdump traceroute conntrack | grep "^\w")

3. Create a package archive:

tar -czf witness-packages.tar.gz *.deb

4. Transfer to witness server (using USB, SCP, or other method)
5. Install packages on witness server:

sudo dpkg -i *.deb

Install all Packages: 

tar -xzf witness-packages.tar.gz

 Fix any missing dependencies (if needed):

sudo apt-get install -f

Troubleshooting

Network Issues

Problem: Unable to reach a gateway or external network.
Solution:

1. Verify that the network interface is up:

ip link show
sudo ip link set <interface> up

2. Check netplan configuration:

sudo netplan --debug apply

3. Verify the routing table:

sudo netplan --debug apply

SSH Connection Issues

Problem: Unable to connect using SSH.
Solution:

1. Verify that the SSH service is running:

sudo systemctl status sshd
sudo systemctl restart sshd

2. Check firewall rules:

sudo ufw status
sudo ufw allow ssh

3. Verify SSH configuration:

sudo sshd -t

chrony Synchronization Issues

Problem: Time is not synchronizing.
Solution:

1. Check chrony status:

sudo systemctl status chrony
sudo chronyc tracking

2. Verify NTP server connectivity:

sudo chronyc sources -v

3. Force time synchronization:

sudo chronyc makestep

User Authentication Issues

Problem: Cannot log in as witness-user.
Solution:

1. Verify that a user account exists:

id witness-user

2. Reset the password if needed:

sudo passwd witness-user

3. Check SSH authorized_keys permissions:

sudo ls -la /home/witness-user/.ssh/