PowerFlex Manager: RKE2 upgrade fails repeatedly due to "405 Not Allowed" error

Summary: PowerFlex Manager: RKE2 upgrade fails repeatedly due to "405 Not Allowed" error while creating the upgrade payload During the PowerFlex Management Platform (PFMP) upgrade, the process fails repeatedly, looping at the point of creating the upgrade payload folder in the http-share. The failure occurs with an HTTP 405 Not Allowed error. ...

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Impacted Version:

Powerflex Manager 4.x

When a user performing an RKE2 upgrade as part of the PFMP upgrade, the following symptoms occur:

  • The upgrade process continuously downloads the bundle but fails during the payload creation phase.
  • Logs from the alcm pod shows the error '405 Not Allowed' while attempting to create the upgrade payload folder in the http-share 
    2024-11-30T06:37:50.452413347-06:00 stdout F 2024-11-30 12:37:50 [pool-2-thread-1] (UpgradePFMP.java:168) [INFO] Starting PFMP upgrade
..
2024-11-30T06:49:06.712045703-06:00 stdout F 2024-11-30 12:49:06 [pool-2-thread-1] (UpgradePFMP.java:718) [DEBUG] PFMP Bundle download progress 100%
..
2024-11-30T06:53:47.628162548-06:00 stdout F 2024-11-30 12:53:47 [pool-2-thread-1] (UpgradePFMP.java:1681) [INFO] Current RKE2 version: 1.25.3
2024-11-30T06:53:47.629703367-06:00 stdout F 2024-11-30 12:53:47 [pool-2-thread-1] (UpgradePFMP.java:1688) [INFO] Available RKE2 version: 1.26.13
...
024-11-30T06:54:02.071950984-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] Moving RKE2 artifacts to upgrade-artifacts folder . . .
2024-11-30T06:54:02.344803044-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] /opt/Dell/ASM/temp/upgrade853246658060560317/Upgrade/prepare-upgrade.sh: line 264: kubectl: command not found
2024-11-30T06:54:02.368496491-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] Creating upgrade payload folder in http-share . . .<html>
2024-11-30T06:54:02.368527723-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] <head><title>405 Not Allowed</title></head>
2024-11-30T06:54:02.368533448-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] <body>
2024-11-30T06:54:02.368548798-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] <center><h1>405 Not Allowed</h1></center>
2024-11-30T06:54:02.368553418-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] <hr><center>nginx/1.21.5</center>
2024-11-30T06:54:02.368565659-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] </body>
2024-11-30T06:54:02.36859845-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG] </html>
2024-11-30T06:54:02.369264313-06:00 stdout F 2024-11-30 12:54:02 [pool-2-thread-1] (ExecuteSystemCommands.java:106) [DEBUG]  done.

 

Cause

PFMP relies on the httpshare-ingress-upload ingress for node-level RKE2 upgrades, enabling the upload of node upgrade modules.
A script "CMO_Patch.zip" provided by the Dell support site restricted http-share upload capabilities to mitigate a vulnerability, resulting in upgrade failures.

 

Resolution

Workaround

  • Ensure that httpshare-ingress-upload ingress does not exist

    kubectl get ingress -A | egrep httpshare-ingress

  • Transfer the YAML File:
    Copy the "httpshare-ingress-upload.zip" file attached to this KB to one of the MVM nodes and extract it.

  • Apply the Ingress Fix:
    Log in to the MVM node, go to the directory where the YAML file is stored, and apply the settings:

    kubectl apply -f httpshare-ingress-upload.yaml

  • Verify the Ingress Creation:
    Ensure the httpshare-ingress-upload ingress has been successfully created. It may take 15-30 seconds for the IP addresses to populate. Check with:

    kubectl get ingress -n powerflex | egrep -i httpshare

  • Restart the alcm pod

    kubectl delete pod $(kubectl get pods -n powerflex | egrep alcm | awk '{print $1}') -n powerflex

  • Check the aclm pod logs to see if the upgrade is running. If not, retry the PFMP Upgrade:
    Start the upgrade process again. The issue should now be resolved.

 

Post-Upgrade Actions:

Once the PFMP upgrade is complete, follow the steps below based on your PFMP version:

  • For systems running PFMP versions earlier than 4.6.1:

    • Reapply the CVE remediation tool (CMO_Patch.zip) to ensure continued security compliance. See KB Remediation for Powerflex Manager CVE-2024-37143 (Dell Support account is required to view this article)
      Note: For any future upgrades, the workaround mentioned above must be applied again unless the system is upgraded to version 4.6.1 or later.

  • For systems running PFMP version 4.6.1 or later:

    • No additional action is required after the upgrade.

 

Fixed In Version: PowerFlex Manager 4.6.1

Additional Information

Important

If the PFMP(Powerflex Manager) installation was previously performed as the root user, ensure that the ownership of the following directories is updated appropriately before proceeding with the PFMP upgrade. For more details, see  PowerFlex Management Platform - PFXM Upgrade Fails On Task Clear landing Dir For Artifacts Destination

Products

PowerFlex rack, PowerFlex Appliance, PowerFlex custom node, PowerFlex Software

Attachments

httpshare-ingress-upload_pkb_en_US_1.zip

Article Properties
Article Number: 000261179
Article Type: Solution
Last Modified: 22 Jan 2025
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.