PowerFlex: How to Power On\Off a Powerflex Appliance Node

Power On\Off a PowerFlex Appliance

• Power on a PowerFlex appliance

To safely power on the system, power on one component at a time, in the order specified in this procedure.

About this task

Power on workflow:

• Power on the PDUs
• Power on the network switches
• Power on the PowerFlex management cluster (ESXi) or PowerFlex management cluster (KVM)
• Power on the CloudLink Center virtual machines that are running on management cluster in VMware vCenter (controller data center)
• For integrated management systems:
  • Power on nodes with PowerFlex management platform
  • Power on nodes with the MDMs
  • Power on all other nodes
• For PowerFlex file (if applicable):
  • For PowerFlex file enabled system with hyperconverged:
    • Power on PowerFlex hyperconverged nodes
    • Activate PowerFlex protection domains
    • Power on PowerFlex file nodes
  • For PowerFlex file enabled system with storage-only:
    • Power on PowerFlex storage-only nodes
    • Activate PowerFlex protection domains
    • Power on PowerFlex file nodes
    • Power on the PowerFlex compute-only nodes

• • For all other PowerFlex systems:
    • Power on the PowerFlex storage-only nodes
    • Power on the PowerFlex hyperconverged nodes with VMware ESXi
    • Activate protection domains

• • Power on the PowerFlex compute-only nodes
  • Power on the VMware NSX Edge nodes (if applicable)
  • Power on all VMs on or single VMware vCenter (customer cluster VMs)
• Check PowerFlex health and rebuild status

Below section is listing only steps for PowerFlex Management Cluster ESXi nodes

For other components and other deployment models See Chapter 23  Dell PowerFlex Appliance with PowerFlex 4.x Administration Guide

• • Power on the PowerFlex management cluster (ESXi)

Use the following procedure to power on the PowerFlex management cluster (ESXi).

Steps

1. Log in to iDRAC and power on all the PowerFlex management nodes. Monitor the virtual console and wait for some time for the VMware ESXi server to appear.

2. Log in to each VMware ESXi server and verify that all the SVMs have started with the host.
3. Power on all the PowerFlex management platform nodes (management virtual machines from VMware ESXi host client):
   1. Log in to VMware ESXi using the host client.
   2. Click Virtual Machines, select the management virtual machine, and click Power on.
   3. Repeat step 3b to power on all the management virtual machines.
4. Log in to the nodes running PowerFlex management platform processes (all three management virtual machines):
   1. To check the status of the rke2-server, type systemctl status rke2-server. Do the following depending on the rke2-server status:

Table  Status of the rke2-server

#systemctl start rke2-server

5. Once the rke2-server is running on all the three PowerFlex management platform nodes, ensure that all nodes are in ready state:

1. 1. Log in to the PowerFlex management platform primary node using SSH and type 

kubectl get nodes

If you see an error message, wait for a few minutes and try again. Once the nodes are in ready state, go to the next step.

6. If you are running PowerFlex management platform 4.6 or lower, do the following:
   1. Restore the cluster monitoring operator (CMO) database:

#alias k="kubectl -n $(kubectl get pods -A | grep -m 1 -E 'platform|pgo|helmrepo' | cut --d' ' -f1)"

kubectl config set-context default --namespace=$(kubectl get pods -A | grep -m 1 -E 'platform|pgo|helmrepo|docker' | cut -d' ' -f1)

#k patch $(k get postgrescluster -o name) --type merge --patch '{"spec":{"shutdown": false}}'

1. 2. To verify the CMO database, type 

echo $(kubectl get pods -l="postgres-operator.crunchydata.com/control-plane=pgo" --no-headers -o name && kubectl get pods -l="postgres-operator.crunchydata.com/instance" --no-headers -o name) | xargs kubectl get -o wide.

7. If you are running a PowerFlex management platform 4.8 or higher:
   1. To gather the IP address of the database service, type

# export NAMESPACE=powerflex

# export DBSA_IP=$(kubectl get service/postgres-monitor-leader -n $NAMESPACE -o jsonpath='{.spec.clusterIP}')

1. 2. To start the database pods and service, type

# curl -i -H "Accept: application/json" "http://$DBSA_IP:8080/pgclusterstartup"

1. 3. To check the status of the startup, type

# curl -i -H "Accept: application/json" "http://$DBSA_IP:8080/pgclusterstartupstatus"

1. 4. Continue to run the status startup command until the status returns verified, see the following example:

HTTP/1.1 200 OK
Date: Mon, 05 Aug 2024 18:36:14 GMT
Content-Length: 47
Content-Type: text/plain; charset=utf-8
{"return_status":0,"startup_status":"verified"}

8. Monitor the PowerFlex management platform status:
   1. To identify the port number for the PowerFlex management platform monitor utility type 

kubectl get services monitor-app -n powerflex -o jsonpath="{.spec.ports[0].nodePort}{\"\n\"}"

Wait for 20-30 minutes and check the overall health status of the PowerFlex management platform.

9. Go to http://<node IP>:port/

Where, the node IP address is a management IP address that is configured on any of the management virtual machines (not the Ingress or PowerFlex Manager IP address).

10. Click PFMP status and wait for all entries to turn green.

Contact Dell Technical Support if the PowerFlex management platform status persists as red, or the main UI is not accessible after 20-30 minutes.

11. Log in to the primary MDM using SSH.
12. Log in to the MDM cluster:

#scli --login --management_system_ip <PFxM_IP_Address> --username admin --password
<PFxM_Password>

13. Verify that the cluster status is Normal:

#scli --query_cluster

Wait for the rebalance to complete before resuming workload. Contact Dell Technical Support if the cluster status is not in Normal state.

14. To check the rebuild or rebalance status, run the following:

#scli --query_all |grep	-i reb

15. Verify that all volumes are available:

#scli --query_all_volumes

16. Power on the vCenter VM from the VMware ESXi host client, if vCenter high availability is configured, power on active, passive, and witness nodes (nodes can start in any order):
    1. Log in to VMware ESXi using the host client.
    2. Click Virtual Machines, select the management virtual machine, and click Power on.
17. Modify the startup order of SVMs to manual to enable the vSphere high availability. This is applicable for all SVMs in the PowerFlex management nodes:
    1. In the vSphere Client, select the host where the VM is located.
    2. Click the Configure tab.
    3. Under Virtual Machines, select VM Startup/Shutdown, and click Edit. The Edit VM Startup and Shutdown window opens.
    4. To modify the startup order of the virtual machines, select a VM from the Automatic Startup category and use the up arrow to move the VM to the Manual Startup category.
    5. Select the SVM and move back to Manual Startup category.
    6. Clear the Automatically start and stop the virtual machines with the system check box and click OK.
    7. Repeat the steps for all the SVMs.
18. Enable vSphere high availability:
    1. Log in to VMware vSphere Client.
    2. Click vSphere Client > Shortcuts > Hosts and Clusters.
    3. Browse to the cluster.
    4. Click the Configure tab.
    5. Select vSphere Availability and click Edit.
    6. Click the toggle button to enable vSphere HA.
    7. Click OK.
19. Power on all other VMs, such as, CloudLink and secure connect gateway:
    1. Log in to VMware vSphere Client.
    2. From vSphere Client > Shortcuts > Hosts and Clusters.
    3. Browse and right-click the VM and click Power > Power on.
20. Verify that all the VMs are up and running.

• Power off a PowerFlex appliance

To safely power off the PowerFlex appliance, power off one component at a time, in the order specified in this procedure.

About this task

Power off workflow:

• Check PowerFlex health, and rebuild status
• Power off all production cluster VMs on VMware vCenter (except the management virtual machine for controller cluster, SVM on both controller and production cluster, and VMware vCenter VM)
• For PowerFlex file:
  • For PowerFlex file enabled system with storage-only:
    • Power off the PowerFlex compute-only nodes with VMware ESXi
    • Power off PowerFlex file nodes
    • Deactivate PowerFlex protection domains
    • Power off PowerFlex storage-only nodes
  • For PowerFlex file enabled system with hyperconverged:
    • Deactivate PowerFlex protection domains
    • Power off PowerFlex hyperconverged nodes
    • Power off PowerFlex file nodes
• For all other PowerFlex systems:
  • Power off the CloudLink Center VMs that are running on management vCenter
  • Deactivate PowerFlex protection domains and power off PowerFlex storage-only nodes (both source and destination protection domains if asynchronous replication is enabled)
  • Power off PowerFlex compute-only nodes with VMware ESXi
  • Power off VMware NSX Edge nodes
    • • Power off PowerFlex storage-only nodes
      • Power off PowerFlex hyperconverged nodes with VMware ESXi
      • Power off the PowerFlex management cluster (ESXi)

• • Power off PDUs

Identify node types in the PowerFlex Manager:

• • • For PowerFlex hyperconverged nodes, click Block > Host or Block > SDSs.
    • For PowerFlex compute-only nodes with VMware ESXi, click Block > Hosts.

• • Power off the PowerFlex management cluster (ESXi)

Power off the PowerFlex management cluster (ESXi) on each of the PowerFlex management cluster VMware ESXi hosts.

Steps

1. Determine the primary MDM IP address:
   1. Log in to PowerFlex Manager to determine the primary MDM.
   2. To view the details of a resource group, click Lifecycle > Resource Groups > PowerFlex management controller 2.0 resource group. Scroll to the Service Details page and make a note of the primary MDM IP address.
2. If you are running PowerFlex management platform 4.6 or lower, do the following:
   1. SSH to any one of the PowerFlex management platform virtual machines and run the following commands to stop the CMO database:

# echo alias k="kubectl -n $(kubectl get pods -A | grep -m 1 -E 'platform|pgo| helmrepo' | cut -d' ' -f1)"

kubectl config set-context default --namespace=$(kubectl get pods -A | grep -m 1 -E 'platform|pgo|helmrepo|docker' | cut -d' ' -f1)

# echo $(kubectl get pods -l="postgres-operator.crunchydata.com/control-plane=pgo" --no-headers -o name && kubectl get pods -l="postgres-operator.crunchydata.com/instance" --no-headers -o name) | xargs kubectl get -o wide

kubectl -n powerflex patch $(kubectl -n powerflex get postgrescluster -o name) --type merge --patch '{"spec":{"shutdown":true}}'

1. 2. Verify the database shutdown. Only the PostgreSQL operator pod pgo must remain the same when running the command:

#echo $(kubectl get pods -l="postgres-operator.crunchydata.com/control-plane=pgo" --no-headers -o name && kubectl get pods -l="postgres-operator.crunchydata.com/instance" --no-headers -o name) | xargs kubectl get -o wide

3. If you are running PowerFlex management platform 4.8 or higher, do the following:
   1. SSH to any of the PowerFlex management platform virtual machines.
   2. To gather the IP address of the database service, type:

# export NAMESPACE=powerflex

# export DBSA_IP=$(kubectl get service/postgres-monitor-leader -n $NAMESPACE -o jsonpath='{.spec.clusterIP}')

1. 3. To shutdown the database pods and service, type:

# curl -i -H "Accept: application/json" "http://$DBSA_IP:8080/pgclustershutdown"

1. 4. To check the status of the shutdown, type

# curl -i -H "Accept: application/json" "http://$DBSA_IP:8080/pgclustershutdownstatus"

1. 5. Continue to run the status shutdown command until the status returns verified, see the following example:

HTTP/1.1 200 OK
Date: Mon, 05 Aug 2024 18:36:14 GMT
Content-Length: 47
Content-Type: text/plain; charset=utf-8
{"return_status":0,"shutdown_status":"verified"}

4. Disable vSphere high availability and change startup order of SVMs. If the host is part of a vSphere high availability cluster, the automatic startup and shutdown of VMs is disabled.

1. 1. Log in to VMware vSphere Client.
   2. From vSphere Client > Shortcuts > Hosts and Clusters.
   3. Browse to the cluster.
   4. Click the Configure tab.
   5. Select vSphere Availability and click Edit.
   6. Click the toggle button to disable vSphere HA.
   7. Click OK.

5. Modify the startup order of SVMs to manual to enable the vSphere high availability. This is applicable for all SVMs in the PowerFlex management cluster:
   1. In the VMware vSphere Client, select the host where the VM is located.
   2. Click the Configure tab.
   3. Under Virtual Machines, select VM Startup/Shutdown, and click Edit. The Edit VM Startup and Shutdown window opens.
   4. Select the Automatically start and stop the virtual machines with the system check box.
   5. Set shutdown action to Guest shutdown.
   6. To change the startup order of VMs, select a virtual machine from the Manual Startup category and use the up arrow to move the VM to the Automatic category.
   7. Select the SVM and move it to Automatic category. This ensures that the SVM automatically stops and starts with the VMware ESXi system.
   8. Repeat the steps for all the SVMs.
   9. Verify the settings and click OK.
6. Gracefully shut down all the VMs except SVMs and vCenter VM:
   1. Log in to VMware vSphere Client.
   2. From vSphere Client > Shortcuts > Hosts and Clusters.
   3. Browse and right-click the VM and select Power > Shut Down Guest OS.

Since SVMs are set to automatically stop and start, there is no need to manually power off.

7. Identify the host where the vCenter server is running and make a note of the IP address or hostname. If vCenter high availability is enabled, make a note of the IP addresses of the hosts where active, passive, and witness nodes are running:
   1. Log in to VMware vSphere Client.
   2. From vSphere Client > Shortcuts > Hosts and Clusters.
   3. Browse the vCenter VM.
   4. Make a note of the host IP address from the Summary page.
8. Shut down the vCenter server VM gracefully. If you have vCenter high availability that is configured, shut down the active, passive, and witness nodes:
   1. Log in to the VMware ESXi using the host client.
   2. Click Virtual Machines > Select the vCenter VM and click to open a browser console.
   3. Press F12 to shut down the VM.
   4. Enter the root password and press OK.
9. Shut down the VMware ESXi servers gracefully:
   1. Log in to VMware ESXi Host Client.
   2. Right-click Host and select Shut down.
   3. To confirm the shutdown of the selected host, click SHUT DOWN.
   4. Repeat the steps for all the remaining nodes.