PowerFlex 4.X: Etcd-databasens langsommelighed forårsager ustabilitetsproblemer i PFMP-klyngen

Resumé:
En udrullet klynge begynder at vise flere pod-genstarter og kan medføre, at noden går i en ikke-klar tilstand, hvilket gør PFMP-brugergrænsefladen utilgængelig.

Scenario:
En PowerFlex Management Platform-instans (PFMP) udrulles på en kundeleveret infrastruktur, der har langsom storage og ikke opfylder storagekravene til PFMP. Symptomerne nedenfor kan identificere dette problem. Dette kan medføre stabilitetsproblemer med K8's klynge- eller Postgres-databasemedlemmer (f.eks. hyppige tilstandsændringer).

I en ny implementering opleves der muligvis ingen problemer, før PowerFlex Manager-automatiseringen udløses til implementering af PowerFlex-klyngen. Forskellige pods kan genstartes flere gange eller gå i en CLBO-tilstand

(Crash Loop Back-off).Nedenfor er de symptomer, der observeres i et produktionsmiljø.

Kør nedenstående kommando for at se hændelserne i klyngen:

kubectl get events
kubectl describe node 

Begivenheder:

   Type     Reason                   Age                From             Message
  ----     ------                   ----               ----             -------
  Normal   RegisteredNode           36m                node-controller  Node pfmp-mvm-cl1-02 event: Registered Node pfmp-mvm-cl1-02 in Controller
  Normal   NodeNotReady             35m                node-controller  Node pfmp-mvm-cl1-02 status is now: NodeNotReady
  Normal   Starting                 33m                kubelet          Starting kubelet.
  Warning  InvalidDiskCapacity      33m                kubelet          invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  33m (x2 over 33m)  kubelet          Node pfmp-mvm-cl1-02 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    33m (x2 over 33m)  kubelet          Node pfmp-mvm-cl1-02 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     33m (x2 over 33m)  kubelet          Node pfmp-mvm-cl1-02 status is now: NodeHasSufficientPID
  Normal   NodeNotReady             33m                kubelet          Node pfmp-mvm-cl1-02 status is now: NodeNotReady
  Normal   NodeAllocatableEnforced  33m                kubelet          Updated Node Allocatable limit across pods
  Normal   NodeReady                33m                kubelet          Node pfmp-mvm-cl1-02 status is now: NodeReady

Nedenfor er det eksempeloutput, der mangler for Kubelet. Logfiler findes i nedenstående sti for hver node:

​​​​​/var/lib/rancher/rke2/agent/logs/kubelet.log

Aug 04 17:12:11 pfmp-mvm-cl1-02 rke2[31392]: time="2023-08-04T17:12:11+09:00" level=debug msg="Wrote ping"
Aug 04 17:12:12 pfmp-mvm-cl1-02 rke2[31392]: E0804 17:12:12.654816   31392 leaderelection.go:367] Failed to update lock: etcdserver: request timed out
Aug 04 17:12:13 pfmp-mvm-cl1-02 rke2[31392]: time="2023-08-04T17:12:13+09:00" level=debug msg="Wrote ping"
Aug 04 17:12:14 pfmp-mvm-cl1-02 rke2[31392]: I0804 17:12:14.526253   31392 leaderelection.go:283] failed to renew lease kube-system/rke2: timed out waiting for the condition
Aug 04 17:12:16 pfmp-mvm-cl1-02 rke2[31392]: time="2023-08-04T17:12:16+09:00" level=debug msg="Wrote ping"
Aug 04 17:12:17 pfmp-mvm-cl1-02 systemd[1]: run-containerd-runc-k8s.io-1ee2f8a41e076afeb4f14eb53d7faea3b1b11c59ecc44aeed0c3ee1333b07a01-runc.DLNAIb.mount: Succeeded.
Aug 04 17:12:23 pfmp-mvm-cl1-02 rke2[31392]: time="2023-08-04T17:12:23+09:00" level=debug msg="Wrote ping"
Aug 04 17:12:26 pfmp-mvm-cl1-02 rke2[31392]: E0804 17:12:26.235602   31392 leaderelection.go:306] Failed to release lock: Operation cannot be fulfilled on configmaps "rke2": the object has been modified; please apply your changes to the latest version and try again
Aug 04 17:12:26 pfmp-mvm-cl1-02 rke2[31392]: time="2023-08-04T17:12:26+09:00" level=fatal msg="leaderelection lost for rke2

etcd loggings:
Kør nedenstående kommando for at finde pod'en for hver etcd og udpak logfilerne:

#  kubectl get pods -n kube-system |grep etcd
etcd-node1                                              1/1     Running     1          92d
etcd-node2                                              1/1     Running     1          92d
etcd-node3                                              1/1     Running     1          92d
# kubectl logs --follow -n kube-system etcd-node1 >> etcd.txt

Log på etcd Logfiler:

 2023-08-04T17:12:12.223766355+09:00 stderr F {"level":"warn","ts":"2023-08-04T08:12:12.223Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"16.840314464s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/operator.tigera.io/installations/\" range_end:\"/registry/operator.tigera.io/installations0\" count_only:true ","response":"","error":"etcdserver: request timed out"}

Alternativt kan de samme logfiler også findes for hver node i nedenstående sti:

 /var/log/pods/kube-system_etcd-_e18aa5e5b83a5a3c56d78e4054612394/etcd

Indvirkning:
Dette kan føre til problemer med PFMP-klyngenodestabiliteten, der får pods til at genstarte flere gange, hvilket medfører, at brugergrænsefladen er utilgængelig.

Ikonet etcd Responstiden er følsom over for langsom storagerespons. Hvis MVM'ernes underliggende storage ikke opfylder kravene, kan der ses problemer med langsom ydeevne inden for etcd miljø. Opbevaringsresponstider på over 1 s kan også medføre forskellige Kubelet proces nedbrud. For eksempel kan lagring bestående af hybriddrev (både HDD og SSD) føre til ydeevneproblemer med etcd.

Nedenstående kommando kan køres for at bekræfte etcd præstation:

for x in $(kubectl get pods -n kube-system |grep etcd |awk '{print $1}') ; do echo "------------------------"; echo $x; echo;kubectl exec -it -n kube-system $x -- etcdctl check perf --cacert="/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt" --cert="/var/lib/rancher/rke2/server/tls/etcd/server-client.crt" --key="/var/lib/rancher/rke2/server/tls/etcd/server-client.key"; echo "------------------------"; echo; sleep 5; done  

Nedenfor er eksempeloutputtet for etcd Fejl i ydeevnen:

for x in $(kubectl get pods -n kube-system |grep etcd |awk '{print $1}') ; do echo "------------------------"; echo $x; echo;kubectl exec -it -n kube-system $x -- etcdctl check perf --cacert="/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt" --cert="/var/lib/rancher/rke2/server/tls/etcd/server-client.crt" --key="/var/lib/rancher/rke2/server/tls/etcd/server-client.key"; echo "------------------------"; echo; sleep 5; done
------------------------
etcd-sio-car-pfmp-mvm-01

 60 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1m0s
PASS: Throughput is 144 writes/s
Slowest request took too long: 1.282354s <<<<<<<<<<<<<<<<<<<<<<<
Stddev too high: 0.151896s
FAIL
command terminated with exit code 1
------------------------

------------------------
etcd-sio-car-pfmp-mvm-02

 60 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1m0s
PASS: Throughput is 150 writes/s
Slowest request took too long: 0.517011s
PASS: Stddev is 0.064465s
FAIL
command terminated with exit code 1
------------------------

------------------------
etcd-sio-car-pfmp-mvm-03

 60 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00% 1m0s
FAIL: Throughput is 138 writes/s
Slowest request took too long: 2.505517s  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Stddev too high: 0.207719s
FAIL
command terminated with exit code 1

Problemet ligger ikke i PFMP-versionen, men i den måde, klynge-VM'erne udrulles på. MVM'ernes underliggende storage skal være placeret lokalt i forhold til den node, hvor disse implementeres, og drevene skal mindst være SSD.

Påvirkede versioner
4. X

Rettet i version
Ingen