Dell NativeEdge: Performance issue in Blueprint Deployment

During blueprint deployment, the Fluent Bit pod experiences high CPU utilization, approaching its defined CPU limit. This elevated usage can slow down the overall blueprint deployment process.

kubectl top po -A --containers | grep fluent

NAMESPACE   POD NAME                  CONTAINER NAME   CPU (cores)   MEMORY (bytes)
hzp         fluentbit-btz2g           fluentbit        999m          43Mi

CPU Usage: 999m means the container is using 999 millicores, which is nearly 1 full CPU core

 Note:- To run the kubectl top command you must install metrics server.   

The root cause of this issue is CPU utilization on fluent bit pod including resources limit not set properly.  

You can optimize CPU limit depends on your resource availability and load. Increase the CPU limit in Daemonset fluent bit. Fluent bit pod is control by Deamon set so you cannot edit the pod directly and must change the CPU limit value in fluent bit Daemonset.

kubectl edit daemonset fluentbit -n hzp
    spec:
      containers:
      - env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: AWS_ACCESS_KEY_ID
              name: eo-management-seaweedfs-s3-secret
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: AWS_SECRET_ACCESS_KEY
              name: eo-management-seaweedfs-s3-secret
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: FLUENT_FOWARD_HOST
          value: localhost
        - name: FLUENT_FOWARD_PORT
          value: "80"
        image: harbor-dell.default.svc.cluster.local/renbaobabga/fluent/fluent-bit:3.2.6
        imagePullPolicy: IfNotPresent
        name: fluentbit
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: "1" >> change it to number depends on your env load and resource allocation  

CPU: "1" >> change it to number depends on your env load and resource allocation. 

Note: - The above change takes some time and re-create the fluent bit pod. 

Verify CPU limit change by running 
kubectl describe po fluentbit-btz2g -n hzp

Verify the CPU use by running 
kubectl top po -A --containers | grep fluent