PowerFlex: How to remove orphaned sdnas-gw NATS consumer

Step 0: Take a VMware snapshot of all 3 MVMs before making changes.

Step 1: Verify the orphan consumer exists:

kubectl exec -it -n powerflex $(kubectl get pods -n powerflex | grep -i "nats-server-box" | awk '{print $1}') -- nats consumer report pfxm-events

Confirm sdnas-gw is listed with a high pending count. If it is not listed, this is a different issue — stop and investigate.

Step 2: Remove the orphaned sdnas-gw consumer:

kubectl exec -it -n powerflex $(kubectl get pods -n powerflex | grep -i "nats-server-box" | awk '{print $1}') -- nats consumer rm pfxm-events sdnas-gw

If this succeeds, skip to Step 3.

If it fails (pending messages can block deletion), continue to Step 2a.

Step 2a: Force removal (only if Step 2 failed):

First, delete the consumer Custom Resource from Kubernetes so the NACK operator will not recreate it after force-removal:

kubectl get consumer -n powerflex
kubectl delete consumer -n powerflex sdnas

Then force-delete the consumer from NATS with the --force flag:

kubectl exec -it -n powerflex $(kubectl get pods -n powerflex | grep -i "nats-server-box" | awk '{print $1}') -- nats consumer rm --force pfxm-events sdnas-gw

If the CR was not deleted before force-removing from NATS, the NACK operator will recreate the consumer. In that case, delete the CR now and restart the NACK pod to resync:

kubectl delete consumer -n powerflex sdnas
kubectl delete pod -n powerflex $(kubectl get pods -n powerflex | grep nats-nack | awk '{print $1}')

Step 3: Purge the accumulated messages:

kubectl exec -it -n powerflex $(kubectl get pods -n powerflex | grep -i "nats-server-box" | awk '{print $1}') -- nats stream purge pfxm-events

Step 4: Verify only 3 consumers remain:

kubectl exec -it -n powerflex $(kubectl get pods -n powerflex | grep -i "nats-server-box" | awk '{print $1}') -- nats consumer report pfxm-events

Expected consumers: alert-service, event-history-external-events, notifier-external-events. No sdnas-gw.

Step 5: Re-run the Message Bus Health diagnostic from the PFxM UI and confirm it returns OK.

Step 6: Monitor for 24-48 hours to confirm messages do not re-accumulate.

Step 7: Once confirmed stable, remove the VMware snapshots.