Solution Builder Extensions for Azure Local -Integrated System AX Server Release Notes
Summary: This article provides information about Solution Builder Extensions (SBEs) for Azure Local. SBEs allow updates to Firmware, Driver, and Windows Defender Application Control (WDAC) supplemental policies for Azure Local clusters running on Dell AX servers. ...
Instructions
Contents
Update Firmware and Drivers Prior to Azure Local Instance Deployment
Manual iDRAC Access Verification
Azure Local instances using Non-Transparent, Unauthenticated Network Proxy
Download Connector – Automated download of future SBE release
Automated Download and Extraction
Installing SBE as Part of Instance Creation
Importing the SBE After Successful Azure Local Instance Creation
Overview
Solution Builder Extensions (SBE) for Azure Local provide firmware, driver, and WDAC supplemental policies for life-cycle management for Azure Local instances running on Dell AX servers.
|
SBE Release Version |
Download links |
Supported Servers |
Driver and Firmware versions |
HCI OS Version |
Supported Solution Version |
|
4.2.2509.1625 |
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=CGKK0
|
AX-660 AX-760
|
Azure Local Support Matrix for 16G (2509) | Solutions for Microsoft Azure |
24H2
23H2 |
12.*.1002.* 12.*.1001.* 11.*.1002.* 11.*.1001.* |
|
AX-4510C AX-4520C |
Azure Local Support Matrix for AX-4500 (2509) | Solutions for Microsoft Azure |
||||
|
4.2.2509.1516 |
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=DTM18
|
AX-650 AX-750 AX-6515 AX-7525
|
Azure Local Support Matrix for 14G-15G (2509) | Solutions for Microsoft Azure
|
24H2
23H2 |
12.*.1002.* 12.*.1001.* 11.*.1002.* 11.*.1001.*
|
|
4.2.2509.1411 |
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=D5R9P
|
AX-640 AX-740xd |
24H2
23H2 |
12.*.1002.* 12.*.1001.* 11.*.1002.* 11.*.1001.*
|
Important Caveats
- NVIDIA ConnectX NIC Resets with driver version 25.1.26647.0: SBE version 4.2.2506 introduced ConnectX driver version 25.1.26647.0 and corresponding firmware. ConnectX NIC resets have been observed by customers under certain workloads. SBE 4.2.2509.n will downgrade the ConnectX driver to version 24.4.26429.0, but the ConnectX firmware must be manually downgraded prior to installing SBE 4.2.2509.n See KB000376360 for details and instructions.
- Dell Micron 7450 and 7500 NVMe SSDs occasionally enter a panic state: Details are described in KB000368482. From the SBE table above, find the version relevant to your product. Install the SBE with updated firmware that resolves the defect for the Micron 7450 and 7500 drives.
New For This Release
SBE 4.2.2509.1625
- Added support for solutions 11.*.1002.* and 12.*.1002.*
- Driver and Firmware 2509 Update
SBE 4.2.2509.1516
- Added support for solutions 11.*.1002.* and 12.*.1002.*
- Driver and Firmware 2509 Update
SBE 4.2.2509.1411
- Added support for solutions 11.*.1002.* and 12.*.1002.*
- Driver and Firmware 2509 Update
Prerequisites
Update Firmware and Drivers Prior to Azure Local Instance Deployment
Drivers and firmware must be installed on servers that will be instance members prior to deploying the instance from the Azure Portal or using an ARM template. This requirement is independent of the SBE installation. The most current driver and firmware versions listed in the support matrix are recommended. The Azure Local support matrix for Dell AX servers is located at the following link:
Support Matrix | Solutions for Microsoft Azure Stack (dell.github.io)
IDRAC Access from Host Operating System
The iDRAC pass-through NIC must be enabled and have an IP address assigned to it. By default, the iDRAC pass-through NIC is enabled and has an APIPA address assigned. This IP address must be accessible by the “Remote NDIS Compatible Device” iDRAC NIC in the host operating system (OS). This is a dedicated link to the host OS and cannot be accessed outside of it.
IDRAC access health checks are introduced starting with December 2024 SBE releases. These health checks verify that that the required iDRAC access has been implemented on each server that is or will be an instance member. The iDRAC access health checks run automatically as part of the standard built-in SBE health checks. These health checks must complete successfully for SBE installation to proceed. Failure remediation instructions are provided if the health check fails.
Manual iDRAC Access Verification
The following procedures provide a manual method for verifying the required iDRAC access. The iDRAC access health checks provide an automated and more comprehensive verification method.
Example
Figure 1: iDRAC passthrough settings
The “Remote NDIS Compatible Device” iDRAC NIC in the host OS is used by the SBE CAUPlug-in to update firmware. The DHCP client should not be disabled on this NIC.
The IP address of the iDRAC NIC can be obtained with the following PowerShell command:
Get-NetAdapter -InterfaceDescription "Remote NDIS Compatible Device" | Get-NetIPAddress -AddressFamily IPv4 | Select-Object IPAddress
Example output:
IPAddress
---------
169.254.0.2
Azure Local instances using Non-Transparent, Unauthenticated Network Proxy
Azure Local instances using non-transparent, unauthenticated network proxy must have the iDRAC pass-through IP address in the network proxy exclusion list. Configuring the non-transparent, unauthenticated network proxy is documented at the following link:
The following example shows how to configure the APIPA address range 169.254.0.0/16 to be in the network proxy exclusion list.
$no_proxy = "localhost,127.0.0.1,.svc,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.contoso.com,node1,node2,ClusterName,169.254.0.0/16"
[Environment]::SetEnvironmentVariable("NO_PROXY", $no_proxy, "Machine")
Download Connector – Automated download of future SBE releases
Overview
The download connector is a component of the SBE that automates the downloading of new SBE bundles as they become available for the instance. Once an SBE that includes the download connector is installed on an instance, manually downloading and importing subsequent SBE versions is no longer required. The update service automates the downloading of the next available SBE for the instance in a manner similar to the automated download of the solution update. The download connector is part of SBE version 4.1.2505.1501 and newer.
SBE versions that include the download connector also include new system and update health checks. These checks proactively verify that the download connector can access the required remote resource when it is time to download the next SBE for the instance. System health checks are run once per day, and update health checks are run during the initial phase of the update process.
Firewall Requirements
The SBE download connector must periodically connect to Internet resources to download SBE bundles for the instance. The following outbound connectivity is required:
|
Endpoint Description |
Endpoint URL |
Port |
|
SBE bundle download |
dl.dell.com/* |
443 |
|
SBE download catalog |
downloads.dell.com/filestore/Prod/SbeDownloadCatalog/AX_SBE_Download_Catalog.xml |
443 |
|
SBE download catalog redirection |
aka.ms/DellAzureLocalSBEDownloadCatalog |
443 |
|
Digicert certificate revocation status |
http://crl3.digicert.com/* |
80 |
|
Digicert certificate revocation status |
http://crl4.digicert.com/* |
80 |
Installation
Download and Extract
The SBE bundle file has the naming convention Bundle_SBE_Dell_AS-HCI-AX-GenerationNumber_VersionNumber.zip.
- Download the applicable SBE bundle using the download link above. SBE bundles are specific to the AX 15G and AX 16G platforms.
- Extract the three SBE files from the SBE bundle zip file. There are two xml files and one zip file. Example:
SBE_Dell_AS-HCI-AX-16G-45n0c _4.2.2506.1600.xmlSBE_Dell_AS-HCI-AX-16G-45n0c _4.2.2506.1600.zipSBE_Discovery_Dell.xml
- Browse to this link and type Ctrl+S to download the
SBE_Discovery_Dell.xmlfile:
https://aka.ms/AzureStackSBEUpdate/DellEMC - Replace the
SBE_Discovery_Dell.xmlfile in the SBE bundle with the file downloaded in step 3.
There are two options for installing the SBE package. The package can be installed as part of instance creation or imported after instance creation is successfully completed. The importing method is also used to update the SBE to a newer version when the current SBE does not include download connector capability.
Automated Download and Extraction
SBE releases starting with 4.1.2505.15n include the download connector that automates downloading and extracting the SBE. Manual download and SBE bundle extraction are unnecessary when an SBE with this capability is installed on the Azure Local instance.
Installing SBE as Part of Instance Creation
Starting with June 2025 Dell golden image release, the SBE files are included in the golden image and already placed in the C:\SBE directory. The following three steps do not need to be performed.
- On each server that will be an instance member, create the following directory:
C:\SBE - Copy the three SBE files to the directory
C:\SBEon each server that will be a instance member - Proceed to deploying the instance from Azure Portal or ARM template
Importing the SBE After Successful Azure Local Instance Creation
- Create a directory on a cluster shared volume (CSV) for importing the SBE.
Example:
C:\ClusterStorage\Infrastructure_1\SBE\SBE-AX-16G_4.1.2412.1202 - Copy the three extracted SBE files to this directory.
- Load the SBE files by running the following command, using the path to the directory containing the three extracted SBE files.
Example:
Add-SolutionUpdate -SourceFolder C:\ClusterStorage\Infrastructure_1\SBE\SBE-AX-16G-45n0c _4.2.2506.1600 - Check the state of the SBE that was just added. The state must be “Ready” in order to proceed.
Example:
Get-SolutionUpdate | Where-Object PackageType -EQ "SBE" - Verify that the Azure Local instance is in a healthy state and is ready for SBE installation by running the following commands on one of the Azure Local instance members:
Get-SolutionUpdate | Where-Object {$_.State -imatch "Ready" -and $_.PackageType -eq "SBE"} | Sort-Object -Property ResourceID -Descending | Select-Object -First 1 | Start-SolutionUpdate -PrepareOnly - Wait for the health check to finish. The solution update state will report “HealthChecking” while the health check is in progress.
- After the health check finishes, run the following command if the solution update state reports “HealthCheckFailed”:
$result = (Get-SolutionUpdate | Where-Object {$_.State -imatch "Ready" -and $_.PackageType -eq "SBE"} | Sort-Object -Property ResourceID -Descending | Select-Object -First 1).HealthCheckResult $result | Where-Object {($_.Status -ne "SUCCESS" -and $_.Severity -eq "CRITICAL" )} | Format-List Title, Status, Severity, Description, Remediation, TargetResourceName, TargetResourceType
Ensure that all tests with “Critical” severity have the “Success” status before continuing to the next step. - Start the SBE installation by running the following command.
Get-SolutionUpdate | Where-Object {$_.State -imatch "Ready" -and $_.PackageType -eq "SBE"} | Sort-Object -Property ResourceID -Descending | Select-Object -First 1 | Start-SolutionUpdate - The SBE installation progress can be checked with the following command:
Get-SolutionUpdate | Where-Object PackageType -EQ "SBE" | Format-List ResourceID, Description, State, SBEVersion, UpdateStateProperties, HealthState
See the following link for more information: https://learn.microsoft.com/en-us/azure/azure-local/update/update-via-powershell-23h2?view=azloc-24113#step-3-import-and-rediscover-updates
Known Issues
Updating Secure Boot component firmware may result in Azure Local registration connection status “Repair Registration Required” when the Azure Local instance is running Solution 10.2503 or newer
Description
Updating Secure Boot component firmware, such as the BIOS, can lead to loss of Azure registration. When this occurs, the Get-AzureStackHCI command shows a connection status of RepairRegistrationRequired.
Background
Azure Local Solution 10.2503 includes mitigation for the Secure Boot bypass vulnerability described in CVE-2023-24932. This mitigation is documented at the link below.
Workaround
This link includes instructions for validating this issue and repairing the Azure registration:
In versions prior to 2505, cumulative OS updates may be unexpectedly installed on the third Tuesday of the month
Description
If the most recently installed update includes an SBE update, cumulative OS updates may be automatically scheduled to install. These cumulative updates are installed on the third Tuesday of each month, beginning at 3am local time. This causes the installed OS version to be newer than the installed Azure Local solution version.
Background
If an SBE uses a CAU plugin which runs the Set-CauClusterRole -CauPluinName $null -Force command, the DaysOfWeek and WeeksOfMonth attributes are added to CauClusterRole. These attributes cause the behavior described above.
Validation
To confirm that an Azure Local instance is affected by this issue, run Get-CauClusterRole. If the output shows the DaysOfWeek and WeeksOfMonth attributes, with values of Tuesday and 3 respectively, the issue is present. Depending on when the issue is detected, cumulative OS updates may not have been installed yet.
Workaround
A detailed workaround can be found in the Microsoft documentation at the following link:
Partially installed SBE version 4.0.0.0 is shown when SBE is installed as part of cluster deployment
DescriptionGet-SolutionUpdateEnvironment reports a partially installed SBE when installing the SBE as part of instance deployment. SBE version 4.0.0.0, shown below, indicates a partially installed SBE.
Get-SolutionUpdateEnvironment
ResourceId : redmond
SbeFamily : AS-HCI-AX
HardwareModel : AX-750
LastChecked : 5/9/2024 12:17:11 AM
PackageVersions : {Solution: 10.2405.0.23, Services: 10.2405.0.23, Platform: 1.0.0.0, SBE: 4.0.0.0}
CurrentVersion : 10.2405.0.23
CurrentSbeVersion : 4.0.0.0
LastUpdated :
State : AppliedSuccessfully
HealthState : Success
HealthCheckResult : {Storage Services Physical Disks Summary, Storage Services Physical Disks Summary, Storage
Services Physical Disks Summary, Storage Services Physical Disks Summary...}
HealthCheck Date : 5/25/2024 11:50:35 PM
AdditionalData : {[SBEAdditionalData, Solution Builder extension is partially installed. Please install the latest
Solution Builder Extension provided by your hardware vendor.
Workaround
Reinstall the same SBE version using the SBE import method after instance deployment successfully finishes.
Background
The SBE CAUPlug-in driver expects all drivers in the SBE payload to be installed on the server during deployment. However, some of the drivers in the SBE payload may not apply to the server model or components in the server model. Microsoft is aware of the limitation and is working to resolve this issue in a future release.
SBE installation does not downgrade drivers that have a newer version than the same driver in the SBE payload
Description
The SBE CAUPlug-in driver does not attempt to downgrade installed drivers that are more recent than the ones in its payload.
Dell supports the driver and firmware versions listed here: Support Matrix | Solutions for Microsoft Azure
SBE packages prior to versions 4.2.2506.1506 and 4.2.2506.1404 do not include the Intel X710 driver in the payload
Description
The SBE CAUPlug-in driver is unable to update the X710 driver because the X710 driver cannot be updated using PNPUtil /Add-driver.
The following command shows the driver version used by the Intel X710 NIC.
Get-NetAdapter -InterfaceDescription *X710* | Format-List Name, DriverInformation
Workaround
- Download the Intel X710 NIC DUP driver package (
Network_Driver_6JHVK_WN64_23.0.0_A00.EXE) from the Dell Support site: https://www.dell.com/support/home/drivers/driversdetails?driverid=6JHVK - Copy
Network_Driver_6JHVK_WN64_23.0.0_A00.EXEto each instance member. - Place one server at a time into maintenance mode and move the virtual machines running on that server to other instance members:
Suspend-ClusterNode -Name <NodeName> -drain - Run
Network_Driver_6JHVK_WN64_23.0.0_A00.EXEon that server to install the driver. Allow the server to reboot if necessary. - Verify the driver version used by the Intel X710 NIC is version 1.19.165.0 and has the date April 05, 2024 by running the following command again:
Get-NetAdapter -InterfaceDescription *X710* | Format-List Name, DriverInformation - Take the server out of maintenance mode and move the virtual machines back to it:
Resume-ClusterNode -Name <NodeName> -Failback - Repeat this procedure for each member of the Azure Local instance.
Install the SBE package once all instance members are using the required X710 driver.
Error event RED149 in iDRAC Lifecycle log after CPLD firmware update
Description
The following error-level event may be logged in the iDRAC Lifecycle log after updating CPLD firmware
Message : Unable to create the job because a CPLD firmware job and non-CPLD job cannot be performed at the same time.
MessageId : IDRAC.2.9.RED149![]()
Workaround
This error can be safely ignored.
Previous Releases