Dell NativeEdge: Manual Witness Node Installation

Purpose

The Witness Node serves as a critical component in the NativeEdge distributed architecture, requiring specific network, security, and time synchronization configurations. After cluster creation, the witness node's LCM (Lifecycle Management) is handled by the cluster master nodes. The steps described in this document are only needed to create a ready-to-be-used Witness node required for NativeEdge 2-Nodes Clusters.

Prerequisites and requirements

Hardware requirements

• Processor: 64-bit x86 processor with minimum 2 cores
• Memory: 4 gigabytes RAM minimum
• Storage: 20 gigabytes available disk space minimum
• Network: Active network interface card

Software requirements

• Ubuntu 24.04 LTS Server ISO image
• Network configuration details (IP address, netmask, gateway, DNS servers)
• Customer-defined password for witness-user account (minimum 8 characters, must contain letters and numbers)

Access requirements

• Physical or remote console access to the target system

Install Ubuntu 24.04 LTS Server

1. Boot from Ubuntu 24.04 LTS ISO

1. 1. Insert the Ubuntu 24.04 LTS Server ISO media or mount it to your virtual machine
   2. Boot the system from the ISO media
   3. At the boot menu, select "Install Ubuntu Server"

2. Select language and keyboard

1. 1. Select your preferred language (default: English)
   2. Select your keyboard layout (default: English (US))
   3. Press Enter to continue

3. Configure network

1. 1. When prompted for network configuration, select "Edit IPv4" on your primary network interface
   2. Choose "Manual" configuration method
   3. Enter the following network details:
      1. Subnet: Enter your subnet in CIDR notation (e.g., 192.168.1.0/24)
      2. Address: Enter the static IP address for this witness node
      3. Gateway: Enter the gateway IP address
      4. Name servers: Enter DNS server addresses (comma-separated)
   4. Select "Save" to apply the configuration

Note: Network configuration can be performed during installation or post-installation. For automated deployment, configure during installation.

4. Configure storage

1. 1. Select "Use an entire disk" for storage configuration
   2. Choose the target disk for installation
   3. Review the storage layout and select "Done"
   4. Confirm the destructive action by selecting "Continue"

5. Profile Setup

1. 1. Enter the following information:
      • Your name: Witness Administrator
      • Your server's name: witness-node (or as per your naming convention)
      • Pick a username: installer (temporary account)
      • Choose a password: Enter a secure temporary password
      • Confirm your password: Re-enter the password

Important: This temporary account will be used only for initial setup. The production account will be created during post-installation configuration.

6. Configure SSH

1. 1. When prompted for SSH setup, select "Install OpenSSH server"
   2. Press Space to enable the checkbox
   3. Do not import SSH identity at this stage
   4. Select "Done" to continue

7. Featured Server Snaps

1. 1. Do not select any additional snaps to maintain a minimal installation
   2. Select "Done" to continue with the installation

8. Complete installation

1. 1. The installation process will begin. This may take 10-15 minutes.
   2. Once complete, select "Reboot Now"
   3. Remove the installation media when prompted
   4. The system will reboot into the newly installed Ubuntu 24.04 LTS Server

Post-Installation Configuration

Log in to the system using the temporary installer account created during installation. Execute the following configuration steps in order:

1. Update system packages

1. 1. sudo apt update
   2. sudo apt upgrade -y 

2. Configure static network (If not done during installation)

If network configuration was not completed during installation, configure it now:

1. 1. Identify your network interface:
      ip link show
   2. Edit the netplan configuration file:
      sudo nano /etc/netplan/00-installer-config.yaml
   3. Configure the static IP address (replace values with your network details):

network:
  version: 2
  ethernets:
    ens160:  # Replace with your interface name
      dhcp4: no
      addresses:
        - 192.168.1.100/24  # Replace with your IP/netmask
      gateway4: 192.168.1.1  # Replace with your gateway
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

1. 4. Apply the network configuration:
      sudo netplan apply
   5. Verify the network configuration:

      ip addr show
      ip route show
      ping -c 4 <gateway-ip>

3. Install required packages
   Note: see section Additional Information if offline installation method is required
   1. Install all necessary packages for the witness node:

sudo apt-get install -y chrony ceph-mon ceph-common wipe iproute2 net-tools iputils-ping iptables ebtables tcpdump traceroute conntrack

1. 2. Verify the installation:

dpkg -l | grep -E 'chrony|ceph-mon|ceph-common'

4. Configure Chrony time synchronization

1. 1. Configure Chrony with your Network Time Protocol servers:

sudo nano /etc/chrony/chrony.conf

1. 2. Add or modify the Network Time Protocol server entries (replace with your organization's Network Time Protocol servers). At least one Network Time Protocol server (with a line starting with server) is mandatory or the witness will be rejected at cluster creation:

# Dell Network Time Protocol Servers
server ntp1.dell.com iburst
server ntp2.dell.com iburst

# Fallback public Network Time Protocol servers
pool 2.ubuntu.pool.ntp.org iburst maxsources 4

# Allow local time to be used if Network Time Protocol servers are unreachable
local stratum 10

# Record the rate at which the system clock gains or loses time
driftfile /var/lib/chrony/drift

# Enable kernel synchronization of the real-time clock
rtcsync

# Step the system clock instead of slew if adjustment is larger than 1 second
makestep 1 3

1. 3. Restart and enable Chrony:

sudo systemctl restart chrony
sudo systemctl enable chrony

1. 4. Verify Chrony synchronization:

sudo chronyc tracking
sudo chronyc sources

5. Create witness user account
   Create the witness user account with a customer-defined password
   Important: This account is temporary and will be used only once by NativeEdge physical nodes during cluster creation initiation. After the cluster is successfully created, this account will be automatically deleted by the NativeEdge system.
   1. Create the witness-user account:

sudo adduser --disabled-password --gecos "Witness Service Account" witness-user

1. 2. Set a custom password for the witness-user account:
      Note: You will need to provide this password during the NativeEdge cluster creation process. Store it securely until cluster initialization is complete. 
      Password Requirements:
      • Minimum length: 8 characters
      • Must contain both letters (a-z, A-Z) and numbers (0-9)
      • Example: Witness2024, Node1234, Edge5678

sudo passwd witness-user

1. 3. Add the user to the sudo group and configure passwordless sudo access:

sudo usermod -aG sudo witness-user

1. 4. Configure passwordless sudo for witness-user:
      Note: This allows witness-user to execute all root commands without password prompts, which is required for NativeEdge cluster initialization.

echo "witness-user ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/witness-user
sudo chmod 440 /etc/sudoers.d/witness-user

1. 5. Generate SSH key pair for witness-user:

sudo -u witness-user ssh-keygen -t rsa -b 4096 -f /home/witness-user/.ssh/id_rsa -N "" -C "witness-user@$(hostname)"

1. 6. Verify SSH key generation:

sudo ls -la /home/witness-user/.ssh/

Expected output:

• • • id_rsa (private key)
    • id_rsa.pub (public key)
    • authorized_keys (for incoming connections)

6. Prepare iptables

1. 1. In the /etc/iptables.rules  with owner being root:root and permissions at 0644, put this content:

*filter
:INPUT DROP [120:8633]
:FORWARD DROP [0:0]
:OUTPUT DROP [2533:382155]

-N LOGGING_INPUT
-N LOGGING_OUTPUT
-N LOGGING_FORWARD
-N CUSTOM_FILTER_INPUT_CHAIN
-N CUSTOM_FILTER_OUTPUT_CHAIN
-N CUSTOM_FILTER_FORWARD_CHAIN

-A INPUT -i lo -j ACCEPT
-A INPUT -s 127.0.0.0/8 -j DROP
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p udp -m multiport --dports 67,68,123,53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m conntrack --ctstate INVALID -j DROP

-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m conntrack --ctstate INVALID -j DROP
-A OUTPUT -p udp -m multiport --dports 67,68,123,53 -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 53 -j ACCEPT

-A INPUT -j CUSTOM_FILTER_INPUT_CHAIN
-A INPUT -j LOGGING_INPUT

-A OUTPUT -j CUSTOM_FILTER_OUTPUT_CHAIN
-A OUTPUT -j LOGGING_OUTPUT

-A FORWARD -j CUSTOM_FILTER_FORWARD_CHAIN
-A FORWARD -j LOGGING_FORWARD

-A LOGGING_INPUT -m limit --limit 10/min -j LOG --log-prefix "IPTables-INPUT-Dropped: " --log-level 4
-A LOGGING_OUTPUT -m limit --limit 10/min -j LOG --log-prefix "IPTables-OUTPUT-Dropped: " --log-level 4
-A LOGGING_FORWARD -m limit --limit 10/min -j LOG --log-prefix "IPTables-FORWARD-Dropped: " --log-level 4
-A LOGGING_INPUT -j DROP
-A LOGGING_OUTPUT -j DROP
-A LOGGING_FORWARD -j DROP
COMMIT

1. 2. In the file /etc/systemd/system/iptables-restore.service  put this content:

[Unit]
Description=Restore iptables rules
Before=network-pre.target
Wants=network-pre.target

[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore /etc/iptables.rules
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

1. 3. Ensure the file /etc/systemd/system/iptables-restore.service ownership is root:root and file permission being 0644

7. Disable Ubuntu disruptive packages and automated updates

1. 1. Run:

apt-get remove -y unattended-upgrades update-notifier-common apport snapd packagekit fwupd ubuntu-pro-client ubuntu-pro-client-l10n
systemctl disable --now apt-daily-upgrade.timer apt-daily.timer dpkg-db-backup.timer fwupd-refresh.timer motd-news.timer
systemctl disable --now ModemManager.service packagekit.service

1. 2. Reboot is recommended to make sure all services are in place and running

Verification

Perform the following verification steps to ensure proper configuration:

1. Network Verification
   1. Verify IP address configuration

ip addr show

1. 2. Verify gateway configuration

ip route show

1. 3. Test network connectivity

ping -c 4 <gateway-ip>
ping -c 4 8.8.8.8

Expected Result: IP address, netmask, and gateway are correctly configured; ping tests succeed.

2. Chrony Verification
   1. Check Chrony service status

sudo systemctl status chrony

1. 2. Verify time synchronization

sudo chronyc tracking

1. 3. Check NTP sources

sudo chronyc sources -v

Expected Result: Chrony service is active and running; system time is synchronized with NTP servers.

3. User Account Verification
   1. Verify witness-user exists

id witness-user

1. 2. Verify user home directory and SSH keys

ls -la /home/witness-user/

1. 3. Verify passwordless sudo configuration

sudo cat /etc/sudoers.d/witness-user

1. 4. Test user login (from another terminal)

su - witness-user

1. 5. Test passwordless sudo (as witness-user)

sudo whoami

Expected Result: User witness-user exists with proper home directory, SSH keys, and can execute sudo commands without password prompts.

4. SSH Server Verification
   1. Verify SSH service status

sudo systemctl status sshd

1. 2. Verify SSH is listening on port 22

sudo ss -tlnp | grep :22

1. 3. Test SSH connection from remote host

ssh witness-user@<witness-node-ip>

Expected Result: SSH service is active and listening; remote SSH connection succeeds.

Optional: Install packages offline 

If the witness server does not have internet access, you can download the required packages on an internet-connected Ubuntu system and transfer them to the witness server:

1. Download packages on internet-connected system:

   apt-get download chrony ceph-mon ceph-common wipe iproute2 net-tools iputils-ping iptables ebtables tcpdump traceroute conntrack

2. Download dependencies:

   apt-get download $(apt-cache depends --recurse --no-recommends --no-suggests --no-conflicts --no-breaks --no-replaces --no-enhances chrony ceph-mon ceph-common wipe iproute2 net-tools iputils-ping iptables ebtables tcpdump traceroute conntrack | grep "^\w")

3. Create a package archive:

   tar -czf witness-packages.tar.gz *.deb

4. Transfer to witness server (using USB, SCP, or other method)
5. Install packages on witness server:
   Extract Packages: 

   sudo dpkg -i *.deb

   Install all Packages: 

   tar -xzf witness-packages.tar.gz

    Fix any missing dependencies (if needed):

   sudo apt-get install -f

Troubleshooting

Network Issues

Problem: Cannot reach gateway or external network
Solution:

1. 1. Verify network interface is up:

ip link show
sudo ip link set <interface> up

1. 2. Check netplan configuration:

sudo netplan --debug apply

1. 3. Verify routing table:

sudo netplan --debug apply

SSH Connection Issues
Problem: Cannot connect via SSH

Solution:

1. 1. Verify SSH service is running:

sudo systemctl status sshd
sudo systemctl restart sshd

1. 2. Check firewall rules:

sudo ufw status
sudo ufw allow ssh

1. 3. Verify SSH configuration:

sudo sshd -t

Chrony Synchronization Issues
Problem: Time not synchronizing

Solution:

1. 1. Check Chrony status:

sudo systemctl status chrony
sudo chronyc tracking

1. 2. Verify NTP server connectivity:

sudo chronyc sources -v

1. 3. Force time synchronization:

sudo chronyc makestep

User Authentication Issues
Problem: Cannot log in as witness-user

Solution:

1. 1. Verify user account exists:

id witness-user

1. 2. Reset password if needed:

sudo passwd witness-user

1. 3. Check SSH authorized_keys permissions:

sudo ls -la /home/witness-user/.ssh/