Application platforms that combine support for both traditional virtual machines (VMs) and lightweight technologies such as containers are of particular interest for organizations that want to get ahead. The transition to containerized network functions is an ongoing process. In the short term, deployment and management of workloads running on VMs is required. Red Hat OpenShift Virtualization unifies the operations and management of VMs and containers on the same platform, helping organizations to benefit from their existing investments in virtualization.

Dell PowerFlex is a software-defined infrastructure that is designed to reduce operational and infrastructure complexity, empowering organizations to move faster by delivering flexibility, elasticity, and simplicity with predictable performance and resiliency at scale. The PowerFlex family provides a foundation that combines compute and high-performance storage resources in a managed, unified fabric. PowerFlex also provides stable and predictable performance for all workloads, including high-performance databases.

This paper provides guidance for the deployment and configuration of a Red Hat OpenShift Virtualization solution using PowerFlex as the underlying software-defined infrastructure. The paper also describes cases such as cloning, live migration, and snapshots of VMs on OpenShift Virtualization along with Dell Container Storage Interface (CSI) for streamlined operations.

Software is the key differentiation in the PowerFlex offering. PowerFlex software components not only provide the distributed software-defined storage services but also help simplify infrastructure management and orchestration. The management software enables comprehensive IT Operational Management (ITOM) and Life Cycle Management (LCM) capabilities that span compute as well as storage infrastructure, from BIOS and Firmware to nodes, software, and networking.

PowerFlex software-defined storage can be deployed in disaggregated (independent compute and storage layers) architectures.

An important component outside of PowerFlex that enables a flexible consumption model for Kubernetes is the PowerFlex CSI driver, developed as a part of the Dell Kubernetes strategy. After loading the CSI driver for PowerFlex into Kubernetes, it can be used to provision persistent volumes from the underlying PowerFlex storage resource. If the Kubernetes deployment is running low on PowerFlex storage resources, you can add PowerFlex storage nodes to increase the system capacity and performance.

The CSI driver connects the PowerFlex system and Kubernetes deployments. It is a storage broker which dynamically provisions volumes from PowerFlex through the PowerFlex API gateway to the Kubernetes cluster. Once the volume is available on PowerFlex, it is immediately mapped to the requesting pod. If a pod is destroyed or rescheduled, the CSI plug-in ensures that the volumes are remapped upon rescheduling of that pod. 

Customers running Kubernetes clusters on PowerFlex use the Dell Container Storage Modules (CSM), which extend the CSI driver capabilities. These modules:

• Provide enterprise storage capabilities to Kubernetes for cloud-native stateful applications.
• Reduce management complexity, so that developers can independently use enterprise storage with ease and automate daily operations.
• Extends storage functionality and capabilities beyond using the CSI driver alone.

These modules include replication, observability, authorization, application mobility, and resiliency.

PowerFlex supports multiple operating systems. PowerFlex is validated with the leading Kubernetes distributions as shown in the following figure:

For this OpenShift Virtualization with PowerFlex solution, a two-layer Dell PowerFlex system provides both compute and storage resources.

Perform the following steps after installing OpenShift Virtualization.

It is recommended that you keep the default scheduling of VMs on the worker node. You can specify the nodes to which you want the workloads, operators, and controllers deployed by configuring node placement rules for the following tasks:

• Deploy VMs only on nodes intended for virtualization workloads.
• Deploy Operators only on infrastructure nodes.
• Maintain separation between workloads.

This solution assumes use of the default scheduling of workloads and does not apply any changes to node placement configuration. If you want to perform node placement, ensure that you adhere to the node placement rules.

Use the operating system images provided by Red Hat to create VMs by one of the following methods:

• Creating a VM from a template by using the web console
• Creating a VM from an instance type by using the web console
• Creating a VM from a VirtualMachine manifest by using the command line

Live migration is the process of moving a running VM to another node in the cluster without interrupting the virtual workload. Live migration enables smooth transitions during cluster upgrades or any time that a node needs to be drained for maintenance or configuration changes.

For live migration, it is recommended to configure a dedicated multus network. In this white paper, VLAN 154 is configured for live migration.

To configure live migration limits and timeouts, go to Virtualization Settings > General Settings > Live migration.

You can perform the same configuration by editing the HyperConverged CR using the following code:

oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
      … 
      liveMigrationConfig:
      allowAutoConverge: false
      allowPostCopy: false
      completionTimeoutPerGiB: 800
      network: nad-154
      parallelMigrationsPerCluster: 5
      parallelOutboundMigrationsPerNode: 2
      progressTimeout: 150
      resourceRequirements:
      vmiCPUAllocationRatio: 10
      uninstallStrategy: BlockUninstallIfWorkloadsExist
      virtualMachineOptions:
      disableFreePageReporting: false
      disableSerialConsoleLog: true
      workloadUpdateStrategy:
      batchEvictionInterval: 1m0s
      batchEvictionSize: 10
      workloadUpdateMethods:
      - LiveMigrate
      workloads: {}
      … 
    

To configure live migration policies for heavy workloads, , see configuration of live migration.

The Virtualization Overview page in the dashboard provides a comprehensive view of virtualization resources, details, status, and top consumers, as shown in the following figure:

This section describes some guidelines and best practices to follow for your PowerFlex and OpenShift cluster.

For the PowerFlex cluster:

• Verify that SDS threads is set to twelve. To check this value using PowerFlex CLI, run the following command on the primary MDM node: scli --set_performance_parameters --all_sds --tech --sds_number_os_threads 12
• Separate the management and data traffic from production application traffic. This separation can be done using VLANs. For more information, see the following documentation:
  • Post-installation activities
  • Dell EMC PowerFlex: Networking Best Practices and Design Considerations.

For OpenShift Virtualization:

• If possible, use the golden images provided by Red Hat when creating VMs. These images are secure images that make the VM configuration process faster and more efficient. See Creating a VM using golden images.
• Monitor VM CPU, memory, and storage consumption in the performance dashboard page to identify potential bottlenecks and optimize resource allocation if necessary.
• Implement a backup and restore strategy for your provisioned VMs to ensure data protection.
• Use the XFS file system for workloads if you want to run databases on OpenShift VMs. Set the default file system as XFS in the CSI values.yaml file and set the default CSI storage class as vxflexos-xfs.
• If possible, configure VM live migration on a separate physical NIC.

This white paper describes the deployment and configuration of Red Hat OpenShift Virtualization using a two-layer Dell PowerFlex system. OpenShift Virtualization enables users to run VMs alongside containers on the same platform, simplifying management and reducing the complexity of maintaining separate infrastructure and management tools.

This paper also describes some VM use cases and demonstrates that over a million IOPS can be achieved in sub millisecond latency using just three OpenShift worker nodes on PowerFlex with the Dell CSI driver.

The following tables show the hardware and software configuration of the storage and compute nodes that are used for the PowerFlex two-layer mode of deployment:

This appendix provides sample YAML files for configuring the network.

The following YAML file is a Bond1 network configuration YAML file. Use this file to create and apply the configuration on the OpenShift cluster.

$ vi bond1-config.yaml
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: bond1-configurations-on-all-nodes
spec:
  desiredState:
    interfaces:
    - name: bond1
       description: Bond with ports ens3f0np0 and ens3f1np1
      type: bond
      state: up
      ipv4:
        dhcp: false
        enabled: false
       link-aggregation:
        mode: 802.3ad
        options:
          miimon: '100'
        port:
        - ens3f0np0
        - ens3f1np1
      mtu: 9000
$ oc apply -f bond1-config.yaml

The following sample YAML file is a VLAN tagging network configuration file. Use this file to create and apply the configuration on the OpenShift cluster

$ vi vlan_tagging_bond1.yaml
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: ocpnode1-bond1-vlans
spec:
  nodeSelector:
    kubernetes.io/hostname: "node3.ocp18.bdcrack13.local"
  desiredState:
    interfaces:
      - name: bond1.151
        description: PowerFlex Data 1 VLAN151 
        type: vlan
        state: up
        ipv4:
          address:
            - ip: 192.168.151.223
              prefix-length: 24
          enabled: true
        vlan:
          base-iface: bond1
          id: 151
        mtu: 9000
      - name: bond1.152
        description: PowerFlex Data 2 VLAN152
        type: vlan
        state: up
        ipv4:
          address:
            - ip: 192.168.152.223
              prefix-length: 24
          enabled: true
        vlan:
          base-iface: bond1
          id: 152
        mtu: 9000 
      - name: bond1.154
        description: VM live migration on VLAN154 
        type: vlan
        state: up
        ipv4:
          address:
            - ip: 192.168.154.223
              prefix-length: 24
          enabled: true
        vlan:
          base-iface: bond1
          id: 154
        mtu: 9000

$ oc apply -f vlan_tagging_bond1.yaml

The following sample YAML file is a bridge interface network configuration on the VLAN interface on one of the nodes. Create and apply a similar configuration to all the nodes by changing the hostname in the kubernetes.io/hostname" parameter.

$ cat bridge_node1_config.yaml
apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: node1-br1-eth1-policy
spec:
  nodeSelector:
     kubernetes.io/hostname: "node3.ocp18.bdcrack13.local"
  desiredState:
    interfaces:
      - name: node1.br154
         description: Linux bridge with bond1.154 as a port for VM live migration
        type: linux-bridge
        state: up
        ipv4:
        - ip: 192.168.154.244
           prefix-length: 24
           enabled: true
        bridge:
           options:
            stp:
               enabled: false
          port:
            - name: bond1.154
$ oc apply -f bridge_node1_config.yaml

The following sample YAML file is a Network Attachment Definition (NAD) file on one of the bridge interfaces. This file will be used to configure the network on VMs. Create a similar configuration for each of the bridge interfaces and apply it to the cluster.

$ vi nad-config-154.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: nad-154
  annotations:
     k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br1
spec:
  config: |
    {
       "cniVersion": "0.3.1",
       "name": "bridge-network",
       "type": "bridge",
       "bridge": "node1.br154",
       "macspoofchk": false,
       "vlan": 154,
       "disableContainerInterface": true,
       "preserveDefaultVlan": false
    }
 
$ oc apply -f nad-config-154.yaml

The following YAML file is a sample VM creation  file:

$ vi vm_create.yaml
  apiVersion: kubevirt.io/v1
 kind: VirtualMachine
 metadata:
  name: rhel-9-minimal
 spec:
  dataVolumeTemplates:
    - metadata:
        name: rhel-9-minimal-volume
      spec:
        sourceRef:
          kind: DataSource
          name: rhel9 
          namespace: openshift-virtualization-os-images 
        storage:  
           resources:
             requests: storage: 64Gi
           storageClassName: vxflexos
  instancetype:
    name: u1.medium 
  preference:
    name: rhel.9 
  running: true
  template:
    spec:
      domain:
        devices: {}
      volumes:
        - dataVolume:
            name: rhel-9-minimal-volume
          name: rootdisk
 
$ oc apply -f vm_create.yaml