Solution Builder Extensions for Azure Local -Integrated System AX Server Release Notes

Contents

• Overview
• Prerequisites
  • Update Firmware and Drivers Prior to Cluster Deployment
  • IDRAC Access from Host OS
  • Azure Local Instances Using Non-Transparent, Unauthenticated Network Proxy
• Download Connector - Automated Download of Future SBE Releases
  • Firewall Requirements
• Installation
  • Dell Support Account Requirements
  • Download and Extract
  • Installing the SBE as Part of Instance Creation
  • Importing the SBE After Successful Instance Creation
• Known Issues
• Previous Releases

Overview

Solution Builder Extensions (SBE) for Azure Local
Firmware, Driver, and WDAC Supplemental policy for life cycle management for HCI OS 23H2 clusters running on Dell AX servers.

Prerequisites

Update Firmware and Drivers Prior to Cluster Deployment

Drivers and firmware must be installed on servers that are instance members before deploying the instance from the Azure Portal or using an ARM Template. This requirement is independent of the SBE installation. The most current driver and firmware versions listed in the support matrix are recommended. The Azure Local support matrix for Dell AX servers is located at the following link: Support Matrix | Solutions for Microsoft Azure Stack (dell.github.io)

IDRAC Access from Host OS

The iDRAC pass-through NIC must be enabled and have an IP address assigned to it. This NIC is enabled by default and has an APIPA address assigned. This IP address must be accessible by the iDRAC NIC Remote NDIS Compatible Device in the host operating system (OS). This is a dedicated link to the host OS and cannot be accessed outside the host OS.

IDRAC access health checks are introduced starting with the December 2024 SBE release. These health checks verify that the required iDRAC access has been implemented on each server that is or will be an instance member. The iDRAC access health checks run automatically as part of the standard SBE health checks. These health checks must complete successfully for SBE installation to proceed. Failure remediation instructions are provided in case the health check fails.

Manual iDRAC Access Verification

The following procedures provide a manual method for verifying the required iDRAC access. The iDRAC access health checks provide an automated and more comprehensive verification method.

Example:

Figure 1: OS to iDRAC Pass-through settings

The iDRAC NIC Remote NDIS Compatible Device in the host OS is used by the SBE CAUPlug-in to update firmware. This NIC must be enabled and have an IP address assigned to it. By default, this NIC is enabled and has an APIPA address assigned to it that can access the iDRAC IP pass-through address. The DHCP client should not be disabled on this NIC.

The IP address of this NIC can be obtained using the following PowerShell command:

Get-NetAdapter -InterfaceDescription "Remote NDIS Compatible Device" | Get-NetIPAddress -AddressFamily IPv4 | Select-Object IPAddress

Example output:

IPAddress
---------
169.254.0.2

Azure Local Instances using Non-Transparent, Unauthenticated Network Proxy

Azure Local instances using non-transparent, unauthenticated network proxy must have the iDRAC pass-through IP address in the network proxy exclusion list. Configuring the non-transparent, unauthenticated network proxy is documented at the following link: Configure proxy settings for environment variables

The following example shows how to configure the APIPA address range 169.254.0.0/16 to be in the network proxy exclusion list:

$no_proxy = "localhost,127.0.0.1,.svc,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.contoso.com,node1,node2,ClusterName,169.254.0.0/16"
[Environment]::SetEnvironmentVariable("NO_PROXY", $no_proxy, "Machine")

Download Connector - Automated Download of Future SBE Releases

Overview

The download connector is a component of the SBE that automates the downloading of new SBE bundles as they become available for the instance. Once an SBE that includes the download connector is installed on your instance, manually downloading and importing follow-on SBE versions is no longer required. The update service automates the downloading of the next available SBE for the instance in a similar manner to the automated download of the solution update. The download connector is part of SBE version 4.1.2505.1501 and newer.

SBE versions that include the download connector also include new system and update health checks. These checks proactively verify that the download connector can access the required remote resource when it is time to download the next SBE for the instance. System health checks are run once per day, and update health checks are run at the initial phase of the update process.

Firewall Requirements

The SBE download connector must periodically connect to Internet resources to download SBE bundles for the instance. The following outbound connectivity is required:

Installation

Download and Extract

The SBE bundle .zip file has the naming convention Bundle_SBE_Dell_AS-HCI-AX-GenerationNumber_VersionNumber.zip.

1. Download the SBE bundle .zip file for your Dell AX server generation using the download link above. The SBE bundles are specific to the AX 15G and AX 16G platforms.

2. Extract the three SBE files from the SBE bundle .zip file. There are two XML files and one .zip file. Example:

   • SBE_Dell_AS-HCI-AX-16G_4.1.2412.1202.xml
   • SBE_Dell_AS-HCI-AX-16G _4.1. 2412.1202.zip
   • SBE_Discovery_Dell.xml
3. Visit this link in a web browser and type Ctrl-S to download the SBE_Discovery_Dell.xml file:
   https://aka.ms/AzureStackSBEUpdate/DellEMC
4. Replace the SBE_Discovery_Dell.xml file in step 2 with the file downloaded in step 3.

There are two options for installing the SBE package. It can be installed as part of instance creation or imported after instance creation is successfully completed. The import method is also used to update the SBE to a newer version.

Installing the SBE as Part of Instance Creation

1. On each server that will be an instance member, create a directory named C:\SBE.
2. Copy the three SBE files to the C:\SBE directory on each server that will be an instance member.
3. Deploy the instance from the Azure Portal or ARM template.

Importing the SBE After Successful Instance Creation

1. Create a directory on a cluster shared volume (CSV) for importing the SBE.

   Example:
   C:\ClusterStorage\Infrastructure_1\SBE\SBE-AX-16G_4.1.2412.1202

2. Copy the three extracted SBE files to this directory.

3. Load the SBE files by running the following command, using the path to the directory that holds the three extracted SBE files.

   Add-SolutionUpdate -SourceFolder C:\ClusterStorage\Infrastructure_1\SBE\SBE-AX-16G_4.1.2412.1202

4. Check the state of the SBE that was added. The State must be Ready.

   Get-SolutionUpdate | Where-Object PackageType -EQ "SBE"

5. Verify that the instance is in a healthy state and is ready for SBE installation by running the following command on one of the instance members:

   Get-SolutionUpdate | Where-Object {$_.State -imatch "Ready" -and $_.PackageType -eq "SBE"} | Start-SolutionUpdate -PrepareOnly

6. Wait for the health check to complete. The solution update state is HealthChecking while the health check is in progress.

7. After the health check completes, run the following commands if the solution update state is HealthCheckFailed:

   $result = (Get-SolutionUpdate | Where-Object {$_.State -imatch "Ready" -and $_.PackageType -eq "SBE"}).HealthCheckResult 
   $result | Where-Object {($_.Status -ne "SUCCESS" -and $_.Severity -eq "CRITICAL" )} | Format-List Title, Status, Severity, Description, Remediation, TargetResourceName, TargetResourceType

8. Ensure that all tests with Critical severity have a status of Success before proceeding to the next step.

9. Start the SBE installation by running the following command.

   Get-SolutionUpdate | Where-Object {$_.State -imatch "Ready" -and $_.PackageType -eq "SBE"} | Start-SolutionUpdate

10. The SBE installation progress can be checked with the following command:

    Get-SolutionUpdate | Where-Object PackageType -EQ "SBE" | Format-List ResourceID, Description, State, SBEVersion, UpdateStateProperties, HealthState

See the following link for more information: Update Azure Local, version 23H2 systems via PowerShell - Azure Local | Microsoft Learn

Known Issues

• Partially installed SBE version 4.0.0.0 is shown when SBE is installed as part of cluster deployment.

  Description
  Get-SolutionUpdateEnvironment reports a partially installed SBE when installing the SBE as part of instance deployment. SBE version 4.0.0.0, shown below, indicates a partially installed SBE.

  Get-SolutionUpdateEnvironment
  
  ResourceId        : redmond
  SbeFamily         : AS-HCI-AX
  HardwareModel     : AX-750
  LastChecked       : 5/9/2024 12:17:11 AM
  PackageVersions   : {Solution: 10.2405.0.23, Services: 10.2405.0.23, Platform: 1.0.0.0, SBE: 4.0.0.0}
  CurrentVersion    : 10.2405.0.23
  CurrentSbeVersion : 4.0.0.0
  LastUpdated       :
  State             : AppliedSuccessfully
  HealthState       : Success
  HealthCheckResult : {Storage Services Physical Disks Summary, Storage Services Physical Disks Summary, Storage
                      Services Physical Disks Summary, Storage Services Physical Disks Summary...}
  HealthCheck Date   : 5/25/2024 11:50:35 PM
  AdditionalData    : {[SBEAdditionalData, Solution Builder extension is partially installed. Please install the latest
                      Solution Builder Extension provided by your hardware vendor.

  Workaround
  Reinstall the same SBE version using the SBE import method after instance deployment successfully completes.

  Background
  The SBE CAUPlug-in driver expects all drivers in the SBE payload to be installed on the server during deployment. However, some of the drivers in the SBE payload may not apply to the server model or components in the server model. Microsoft is aware of the limitation and is working to resolve this issue in a future release.

• SBE installation does not downgrade drivers that have a newer version than the same driver in the SBE payload.

  Description
  The SBE CAUPlug-in driver does not attempt to downgrade installed drivers that are more recent than the ones in its payload.

  Dell supports the driver and firmware versions listed here: Support Matrix | Solutions for Microsoft Azure.

• The SBE no longer includes the Intel X710 driver in the payload.

  Description
  The SBE CAUPlug-in driver is unable to update the X710 driver because the X710 driver cannot be updated using PNPUtil /Add-driver.

  The following command shows the driver version used by the Intel X710 NIC.

  Get-NetAdapter -InterfaceDescription *X710* | Format-List Name, DriverInformation

   

  Workaround

  1. Download the Intel X710 NIC DUP driver package (Network_Driver_6JHVK_WN64_23.0.0_A00.EXE) from the Dell Support site:
     Intel NIC Family Version 23.0.0 Drivers for Windows 64-bit Operating Systems | Driver Details.

  2. Copy Network_Driver_6JHVK_WN64_23.0.0_A00.EXE to each instance member.

  3. Place one server at a time into maintenance mode and move the virtual machines running on that server to other instance members:

     Suspend-ClusterNode -Name <NodeName> -drain

  4. Run Network_Driver_6JHVK_WN64_23.0.0_A00.EXE on that server to install the driver. Allow the server to reboot if necessary.

  5. Verify the driver version used by the Intel X710 NIC is version 1.19.165.0 and has the date April 05, 2024 by running the following command again:

     Get-NetAdapter -InterfaceDescription *X710* | Format-List Name, DriverInformation

  6. Take the server out of maintenance mode and move the virtual machines back to it:

     Resume-ClusterNode -Name <NodeName> -Failback

  7. Repeat the procedure for each member of the Azure Local instance.

  8. Install the SBE package once all instance members are using the required X710 driver.

• Error event RED149 in iDRAC Lifecycle log after CPLD firmware update.

            Description

The following error-level event may be logged in the iDRAC Lifecycle log after updating CPLD firmware:

Message         : Unable to create the job because a CPLD firmware job and non-CPLD job cannot be performed at the same time.
MessageId       : IDRAC.2.9.RED149

Workaround This error event can be safely ignored. 

• On versions prior to 2505, cumulative OS updates may be unexpectedly installed on the third Tuesday of the month.
   

           Description

If the most recently installed update includes an SBE update, cumulative OS updates may be automatically scheduled to install. These cumulative updates are installed on the third Tuesday of each month, beginning at 3am local time. This causes the installed OS version to be newer than the installed Azure Local solution version.
 

Background

If an SBE uses a CAU plugin which runs the Set-CauClusterRole -CauPluinName $null -Force command, the DaysOfWeek and WeeksOfMonth attributes are added to CauClusterRole. These attributes cause the behavior described above.
 

Validation

To confirm that an Azure Local instance is affected by this issue, run Get-CauClusterRole. If the output shows the DaysOfWeek and WeeksOfMonth attributes, with values of Tuesday and 3 respectively, the issue is present. Depending on when the issue is detected, cumulative OS updates may not have been installed yet.
 

Workaround

Case1: Cumulative OS updates have not yet been installed unexpectedly.

Run the following on any node to check for and remove the CAU trigger. This code should be run while logged in as the deployment user. It ensures that an update is not in progress before removing the trigger.

$entry = Get-CauClusterRole -ErrorAction SilentlyContinue | where-object { $_.Name -eq "DaysOfWeek" }  
if ($null -ne $entry -and $entry.Value -eq "4") {
     Write-host "CauClusterRole is scheduled to trigger on 3rd Tuesday!"
     # before we remove, make sure no update is running
     $updateId = (Get-SolutionUpdate | ?{$_.State -like "*ing"}).ResourceId
     if ($null -ne $updateId) {
         throw "Unable to remove scheduled CAU trigger - an update is in progress:`n$($updateId)"
     }
     Remove-CauClusterRole -Force -ErrorAction SilentlyContinue 3>$null 4>$null
     $entry = Get-CauClusterRole -ErrorAction SilentlyContinue | where-object { $_.Name -eq "DaysOfWeek" }
     if ($null -ne $entry -and $entry.Value -eq "4") {
         throw "Attempt to call 'Remove-CauClusterRole' failed.  Assure you are logged in as the deployment user."
     }
     else {
         Write-Host "Confirmed removal of scheduled CAU run!"
     }
}
else {
     Write-Host "CauClusterRole already removed or not scheduled to trigger automatically"
}

Case 2: At least one cumulative OS update has already been installed unexpectedly.

Run the following to confirm that the unexpected updates were caused by this issue:

$getCauReportBlock = {
    [array]$allReports = Get-CauReport -Detailed
    $results = @()
    foreach ($report in $allReports) {
        $summaryReport = @{}
        $summaryReport.RunId = $report.ClusterResult.RunId.Guid
        $summaryReport.StartTimestamp = $report.ClusterResult.StartTimestamp
        $summaryReport.Plugin = $report.Plugin  
        $results += $summaryReport
    }
    return ($results | ?{$_.Plugin -like "*Microsoft.WindowsUpdatePlugin*"})
}
# type the deployment user credentials when prompted
Invoke-Command -Credential $null -Authentication Credssp -Computername localhost -ScriptBlock $getCauReportBlock

A CAU run which occurred on the third Tuesday of the month and used the Microsoft.WindowsUpdate plugin is an indicator of this issue. Run the code in Case 1 above to remove the CAU trigger. When it is feasible to do so, update the cluster using the normal process in order to match the Azure Local and OS build versions.
 

Further information

More information about this issue, including screenshots of the output of the above commands, can be found at this link. 

Previous Releases