Recommendations and considerations when using a Kubernetes cluster
Review the following information that is related to the deployment, configuration, and use of the Kubernetes cluster as an asset source in
PowerProtect Data Manager:
Performance and scale limits
The following limits have been tested successfully with
PowerProtect Data Manager.
NOTE:These numbers are not maximum or hard limits, but should be considered when scaling your environment.
Table 1. Configuration and tested limitsKubernetes cluster tested limits
Configuration
Tested limits
Number of namespaces in a cluster
300, with 1,000 resources per namespace
Number of vSphere CSI PVCs in a cluster
250
Enable concurrent multi-streaming in File System agent for protecting large volumes
The File System agent is the default data mover that
PowerProtect Data Manager uses for protecting persistent volumes that are provisioned by CSI drivers other than the vSphere Container Storage Interface (CSI) driver. For volumes larger than 100 GB,
PowerProtect Data Manager can protect PVCs using multiple File System agent streams. To enable this functionality, you can add the annotation
ppdm.config.fsagent/streaming_mode: concurrent to the PVC, which the controller then passes to the cProxy. Adding this annotation can improve performance and reduce backup latency by up to 50%.
The number of required streams is calculated dynamically, and will not exceed eight streams to avoid impact to CPU performance. Configuration of the chunk size impacts the stream count calculation.
NOTE:Multiple streams are enabled only for sub-trees with a size greater than the chunk size. No chunking occurs if the volume is a root folder with no sub-folders that contains only flat files. The default chunk size is 50 GB.
Add line to the
custom-ports file when not using port 443 or 6443 for the Kubernetes API server
If a Kubernetes API server listens on a port other than 443 or 6443, an update is required to the
PowerProtect Data Manager firewall to allow outgoing communication on the port being used. Before you add the Kubernetes cluster as an asset source, perform the following steps to ensure that the port is open:
Log in to
PowerProtect Data Manager, and change the user to
root.
Add a line to the file
/etc/sysconfig/scripts/custom-ports that includes the port number that you want to open.
Run the command
service SuSEfirewall2 restart.
This procedure should be performed after a
PowerProtect Data Manager update, restart, or server disaster recovery.
Log locations for Kubernetes asset backup and restore operations and pod networking
All session logs for Kubernetes asset protection operations are pulled into the
/logs/external-components/k8s folder on the
PowerProtect Data Manager host.
PVC parallel backup and restore performance considerations
To throttle system performance, the default settings for
PowerProtect Data Manager data protection are five parallel namespace backups and two parallel namespace restores per Kubernetes cluster. You can change the
ppdm.backup.concurrency and
ppdm.restore.concurrency settings in the controller configuration for the Kubernetes cluster asset source, as described in the section
Controller Configurations. PVCs within a namespace are backed up and restored sequentially. Refer to the section
Resource requirements of PowerProtect Data Manager components on Kubernetes cluster to determine the overhead on the Kubernetes cluster with the higher concurrency settings before making any changes.
You can queue up to 100 namespace backups across protection policies in
PowerProtect Data Manager.
Resource requirements of
PowerProtect Data Manager components on Kubernetes cluster
At any time during backup, with the default backup and restore concurrency settings the typical footprint of
PowerProtect Data Manager components (Velero, PowerProtect Controller, cProxy) is less than 2 GB RAM Memory and four CPU cores. This resource utilization only peaks during backups.
The following resource limits are defined on the PowerProtect PODs, which are part of the
PowerProtect Data Manager stack:
Velero maximum resource usage: 1 CPU core, 256 MB memory
PowerProtect Controller maximum resource usage: 1 CPU core, 256 MB memory
PowerProtect cProxy pods: Each cProxy pod typically consumes less than 300 MB memory and less than 0.8 CPU cores. These pods are created and terminated within the backup and restore job.
One cProxy pod is started for each namespace with PVCs that is backed up or restored.
Only Persistent Volumes with VolumeMode Filesystem supported
Backup and recovery of Kubernetes cluster assets in
PowerProtect Data Manager is only supported for Persistent Volumes with the VolumeMode
Filesystem.
Objects using PVC scaled down before starting the restore
The following activities occur before a PVC restore to the original namespace or an existing namespace when
PowerProtect Data Manager detects that the PVC is in use by a Pod, Deployment, StatefulSet, DaemonSet, ReplicaSet, or Replication Controller:
PowerProtect Data Manager scales down any objects using the PVC.
PowerProtect Data Manager deletes the daemonSet and any Pods using PVCs.
Upon completion of the PVC restore, any objects that were scaled down are scaled back up, and any objects that were deleted are re-created. Ensure that you shut down any Kubernetes jobs that actively use the PVC before running a restore.
NOTE: If
PowerProtect Data Manager is unable to reset the configuration changes due to a controller crash, it is recommended to delete the Pod, Deployment, StatefulSet, DaemonSet, ReplicaSet, or Replication Controller from the namespace, and then perform a Restore to Original again on the same namespace.
Restore to a different existing namespace can result in mismatch between UID of pod and UID persistent volume files
A
PowerProtect Data Manager restore of files in persistent volumes restores the UID and GID along with the contents. When performing a restore to a different namespace that already exists, and the pod consuming the persistent volume is running with restricted Security Context Constraints (SCC) on OpenShift, the UID assigned to the pod upon restore might not match the UID of the files in the persistent volumes. This UID mismatch might result in a pod startup failure.
For namespaces with pods running with restricted SCC, it is recommended to use one of the following restore options:
Restore to a new namespace where
PowerProtect Data Manager restores the namespace resource as well.
Restore to the original namespace if this namespace still exists.
Restore considerations for OpenShift
When restoring to a new namespace or an existing namespace, the restored service accounts are not added to any SCC on OpenShift. After the restore completes, run the following command to add the restored service accounts to SCC as required:
oc adm policy add-scc-to-user
scc name -z
service account name -n
restored namespace
Data is not available for the Topic
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please provide ratings (1-5 stars).
Please select whether the article was helpful or not.
Comments cannot contain these special characters: <>()\