If you have been following this series, you already know how I built my Azure Local demolab step by step: preparing the Hyper-V host, configuring the domain controller, registering the node with Azure Arc and then walking through the portal deployment. That workflow works, but it is entirely manual. Every time I tear down and rebuild the lab I repeat the same sequence of clicks and commands and every time I do that I risk a small configuration drift.

A few months ago I decided to change that. The goal was straightforward: replace the portal deployment path with a fully automated Terraform run that I could trigger from a pipeline with a service account and that I could later reuse for AVD, AKS and other workloads on top of Azure Local. Less clicking, more repeatable infrastructure.

Getting there turned out to be more work than I expected. The Azure Verified Module (AVM) for Azure Local is a good starting point, but it was written against a specific point in time and the Azure Local deployment API has moved since then. From the end of 2025 onward, several things changed: resource provider behavior, required role assignments and the API version that the Azure control plane actually accepts. When I first tried to apply the module against my lab, I got a series of failures that took real investigation to understand.

This article covers what I built, what broke, how I fixed it and how the deployment now flows end to end. I will also explain where I plan to take the repository from here.

The AzSHCI Repository

All of the automation lives in my AzSHCI repository.

It has two parallel paths that work together:

• scripts/01Lab/: PowerShell scripts that handle everything from Azure prerequisites through Hyper-V infrastructure setup, domain controller configuration and Arc registration. These run before Terraform comes into the picture.
• terraform/: A root Terraform configuration that calls a local fork of the AVM module. The fork carries the fixes and additions that were needed to make the deployment work against the current Azure API.

The long-term vision for the repository is a single codebase that can deploy not just the Azure Local cluster but also the workloads on top of it: AVD host pools, AKS clusters and potentially other services. The Terraform path is designed from the start to be consumed from a CI/CD pipeline using a service principal, so every credential is handled through a Key Vault and no secrets live in the repository.

Prerequisites: The First Script

Before any Terraform runs, Azure needs to be in the right state. The script scripts/01Lab/00_AzurePreRequisites.ps1 handles that in a single, interactive run.

What it does:

1. Checks for and installs the required Az PowerShell modules (Az.Accounts, Az.Resources).
2. Verifies your Azure session and prompts a device code login if no active session is found.
3. Lets you select a subscription interactively.
4. Lets you choose an existing resource group or create a new one.
5. Assigns the required RBAC roles to a user or to a newly created service principal.
6. Registers all required resource providers.

The roles it assigns fall into two scopes. At resource group scope: Azure Connected Machine Onboarding, Azure Connected Machine Resource Administrator, Key Vault Data Access Administrator, Key Vault Secrets Officer, Key Vault Contributor and Storage Account Contributor. At subscription scope: Azure Stack HCI Administrator and Reader.

The resource providers it registers cover the full Azure Local stack: Microsoft.HybridCompute, Microsoft.AzureStackHCI, Microsoft.Kubernetes, Microsoft.KubernetesConfiguration, Microsoft.ExtendedLocation, Microsoft.ResourceConnector, Microsoft.HybridContainerService and several others.

If you create a service principal through this script, it prints the connection details at the end. Those credentials go into your Terraform variables file and Key Vault reference and from that point the pipeline can run unattended. Here is a full run with sensitive values redacted:

Checking required Az modules...
Module 'Az.Accounts' is available.
Module 'Az.Resources' is available.
All required modules loaded.
Active session: admin@<tenant>.com on subscription 'Azure-Abonnement 1'.
Use this session? (Y/N): N
Starting device code login...
[Login to Azure] To sign in, use a web browser to open the page https://login.microsoft.com/device
and enter the code GAX*****QB to authenticate.

Authenticated to Azure.
Retrieving available subscriptions...

Select the subscription to use:
    0. Azure-Abonnement 1
Select subscription: 0
Using subscription 'Azure-Abonnement 1' (<subscription-id>).

Resource Group setup:
  1. Use an existing resource group
  2. Create a new resource group
Select option (1 or 2): 2

Recommended resource group name: 'rg-azlocal-lab'
Enter resource group name (press Enter to accept recommendation): rg-azlocal-demolab

Select the Azure region for the new resource group:
    0. westeurope
    1. northeurope
    2. eastus
    ...
Enter a list number or type a region name directly: 0
Creating resource group 'rg-azlocal-demolab' in 'westeurope'...
Resource group 'rg-azlocal-demolab' created.

Select how to assign the required Azure RBAC roles:
  1. Assign to an existing user account
  2. Create a new Service Principal and assign roles to it
Select option (1 or 2): 2

Recommended SPN name: 'sp-azlocal-lab'
Enter SPN display name (press Enter to accept recommendation): sp-azlocal-demolab
Creating app registration 'sp-azlocal-demolab'...
App registration created. AppId: <app-id>
Creating service principal...
Service principal created. ObjectId: <object-id>
Generating client secret (valid for 2 years)...
Client secret generated.
Waiting 20 seconds for the SPN to propagate before assigning roles...

Assigning resource group scoped roles to 'sp-azlocal-demolab'...
Assigned 'Azure Connected Machine Onboarding' at RG scope.
Assigned 'Azure Connected Machine Resource Administrator' at RG scope.
Assigned 'Key Vault Data Access Administrator' at RG scope.
Assigned 'Key Vault Secrets Officer' at RG scope.
Assigned 'Key Vault Contributor' at RG scope.
Assigned 'Storage Account Contributor' at RG scope.

Assigning subscription scoped roles to 'sp-azlocal-demolab'...
Assigned 'Azure Stack HCI Administrator' at subscription scope.
Assigned 'Reader' at subscription scope.

Checking and registering required resource providers...
Provider 'Microsoft.HybridCompute' is already registered.
Provider 'Microsoft.GuestConfiguration' is already registered.
Provider 'Microsoft.HybridConnectivity' is already registered.
Provider 'Microsoft.AzureStackHCI' is already registered.
Provider 'Microsoft.Kubernetes' is already registered.
Provider 'Microsoft.KubernetesConfiguration' is already registered.
Provider 'Microsoft.ExtendedLocation' is already registered.
Provider 'Microsoft.ResourceConnector' is already registered.
Provider 'Microsoft.HybridContainerService' is already registered.
Provider 'Microsoft.Attestation' is already registered.
Provider 'Microsoft.Storage' is already registered.
Provider 'Microsoft.KeyVault' is already registered.
Provider 'Microsoft.Insights' is already registered.

Azure prerequisites setup completed.

Summary:
  Subscription : Azure-Abonnement 1 (<subscription-id>)
  Resource Group: rg-azlocal-demolab (location: westeurope)
  Principal    : sp-azlocal-demolab

================================================================
  SERVICE PRINCIPAL CONNECTION DETAILS
  Save these values securely. The secret cannot be retrieved
  again after this session ends.
================================================================

  Display Name    : sp-azlocal-demolab
  Tenant ID       : <tenant-id>
  Subscription ID : <subscription-id>
  App ID          : <app-id>
  Client Secret   : <client-secret>
  Secret Expiry   : 2028-04-17

  IMPORTANT: Rotate this secret before it expires to avoid service disruptions.

  To connect with this SPN in PowerShell:

  $spnCredential = New-Object PSCredential(
      "<app-id>",
      (ConvertTo-SecureString "<client-secret>" -AsPlainText -Force))
  Connect-AzAccount -ServicePrincipal `
      -Tenant "<tenant-id>" `
      -Subscription "<subscription-id>" `
      -Credential $spnCredential

================================================================

Infrastructure and Cluster Preparation

With the Azure side ready, the next scripts prepare the local environment:

• 00_Infra_AzHCI.ps1 builds the Hyper-V infrastructure: virtual switches, storage paths and the Azure Local node VM.
• 01_DC.ps1 sets up the domain controller that the cluster needs for Active Directory integration.
• 02_Cluster.ps1 performs the Arc registration of the node. After this script completes, the machine appears in Azure as an Arc-enabled server and is ready for the Terraform deployment step.

The Terraform Architecture

The Terraform configuration is a thin root module that creates a few shared prerequisites and then calls the Azure Local cluster module. The shared prerequisites are:

• Key Vault with RBAC authorization enabled, used to store the deployment credentials.
• Witness storage account for the cluster quorum.
• Required role assignments at resource group scope for the Arc machine identity and the Azure Stack HCI resource provider service principal.
• Edge device registration (Microsoft.AzureStackHCI/edgeDevices) for the Arc node.

The cluster module is a local fork of the AVM module. The fork is not a divergence for its own sake. It carries specific fixes that the upstream module did not have at the time of writing and I will describe those in the next section.

Staged deployment

The deployment happens in two stages, controlled by a single variable:

Stage 1 (is_exported = false): Terraform creates the Key Vault, storage account, RBAC assignments and edge device registration, then submits deploymentSettings to Azure with deploymentMode = Validate. Azure runs a validation sequence that checks connectivity, Active Directory and node configuration. This takes roughly 10 to 30 minutes.

Stage 2 (is_exported = true): After validation succeeds in the portal, you flip is_exported to true and run terraform apply again. This patches deploymentMode to Deploy and Azure starts the full cluster provisioning, which takes 30 to 60 minutes.

A second variable, deployment_completed, controls whether Terraform attempts to read post-deployment data sources like the custom location. Set it to false during and after deployment and flip it to true only after the Azure portal confirms the deployment is complete.

What Broke and How I Fixed It

This section documents the real debugging journey. I am including it because if you try to use any version of the AVM module against a current Azure subscription, you will likely hit some of these same issues.

Missing edgeDevices resource

The most impactful missing piece was the Microsoft.AzureStackHCI/edgeDevices resource. This resource registers each Arc machine with the HCI edge management system and lets the LcmController’s ARM client communicate back to Azure. Without it, every deployment settings validation failed with:

Failed to download deployment settings file using edge Arm client

The upstream AVM module did not include this resource at all. Adding edgedevices.tf with the correct API version (2025-09-15-preview) and ensuring it depends on the RBAC assignments resolved the failure.

Missing role assignments

The ARM QuickStart template assigns two roles to the Arc machine identity at resource group scope that the AVM module was not creating:

• Azure Stack HCI Device Management Role
• Azure Stack HCI Connected InfraVMs

Without these, the LcmController cannot install the required Arc extensions. I added machine_rg_role_assign in rolebindings.tf to mirror the ARM template behavior.

API version changes

Between late 2024 and early 2025, the live Azure endpoint stopped accepting networkingType and networkingPattern as body fields. Sending them causes an HTTP 400 ObjectAdditionalProperties error. The fix is to omit them by setting both variables to empty strings; the merge() logic in locals.tf then drops them from the JSON body.

The API version itself also matters. The ARM QuickStart template uses 2025-09-15-preview for both Microsoft.AzureStackHCI/clusters and Microsoft.AzureStackHCI/clusters/deploymentSettings. Using a newer preview version can change how the control plane processes the request. After several failed attempts with 2026-03-01-preview, reverting to 2025-09-15-preview was the right call.

The LcmController 0.settings bug

This one took the most iterations to nail down and I want to be honest: I went down several wrong paths before finding the real cause. The full investigation trail, including the false leads and the intermediate workarounds I tried, is documented in the CHANGELOG if you want the unfiltered version. I will keep this section to what actually matters.

The symptom was a BITS failure during validation:

Cannot bind argument to parameter 'Source' because it is an empty string.

The root cause is a type-checking bug in DownloadHelpers.psm1 inside the AzureEdgeLifecycleManager Arc extension (NuGet package 10.2601.0.1162). When Terraform creates deploymentSettings, Azure writes a minimal object to the LcmController runtime settings file that carries only cloud identity markers and no actual deployment payload. A null-check in GetTargetBuildManifest fails to detect this correctly because [string]::IsNullOrEmpty() returns $false for any non-null PowerShell object, even one with no useful content. The fallback that would download the cloud manifest never activates and the code ends up with four empty download URLs.

One important lesson from this: the four required Arc extensions (AzureEdgeTelemetryAndDiagnostics, AzureEdgeDeviceManagement, AzureEdgeLifecycleManager, AzureEdgeRemoteSupport) must be installed through the cluster creation process itself. Trying to pre-stage them manually before terraform apply is not only unnecessary, it can leave the node in a state where validation rejects it. Let Terraform handle the extension installation as part of Stage 1. The 03_TroubleshootingExtensions.ps1 script remains in the repository as a troubleshooting tool for environments where extensions end up in a Failed state after a previous failed apply, not as a required step in the normal flow.

Step-by-Step Deployment

With all of the above in place, here is the actual deployment flow.

Step 1: Azure prerequisites

Run 00_AzurePreRequisites.ps1, select or create a resource group and either assign roles to your own account or let the script create a service principal. Note the SPN credentials if you created one. The full output of this script is shown in the Prerequisites section above.

Step 2: Build the Hyper-V infrastructure

Run 00_Infra_AzHCI.ps1 to create the Hyper-V host infrastructure for the lab node. This step is identical to the one described in the demolab article, so refer to it for a detailed walkthrough.

Step 3: Configure the domain controller

Run 01_DC.ps1 to set up Active Directory on the domain controller VM. Again, this step is identical to the one covered in the demolab article.

Step 4: Register the node with Arc

Run 02_Cluster.ps1 to Arc-register the Azure Local node. Confirm the machine appears in Azure portal as an Arc-enabled server before continuing.

Step 5: Configure Terraform variables

Copy terraform/terraform.tfvars.example to terraform/terraform.tfvars. Replace every TODO value with your actual credentials and network settings. Key variables for a single-node lab:

management_adapters = ["MGMT1"]
rdma_enabled        = false
networking_type     = ""
networking_pattern  = ""
witness_type        = ""
is_exported         = false
deployment_completed = false

Step 6: Authenticate with Azure CLI

Terraform uses the Azure CLI session to authenticate. Log in with the service principal created in Step 1 and set the target subscription:

az login --service-principal `
    --username "<app-id>" `
    --password "<client-secret>" `
    --tenant "<tenant-id>"
az account set --subscription "<subscription-id>"

Step 7: Initialize and run Stage 1 (Validate)

terraform init
terraform plan
terraform apply

Review the plan output before confirming the apply. Terraform creates the Key Vault, storage account, RBAC assignments and edge device registration, then submits the deployment settings to Azure for validation.

Validation runs as part of the apply itself. Once Terraform reports the apply as complete, Stage 1 is done (takes arround 90 minutes) and you can move straight to Stage 2.

Step 8: Switch to Stage 2 (Deploy)

Set is_exported = true in terraform.tfvars and run terraform apply again. Terraform patches deploymentMode to Deploy and Azure starts the full cluster provisioning. This apply will run for 90 to 180 minutes while Azure works through the FullCloudDeployment plan.

While Terraform is waiting, you can follow the deployment progress in real time in the Azure portal under the Azure Local resource. Each step of the plan is shown individually, which makes it much easier to spot a failure early rather than waiting for a Terraform timeout.

Once the apply finishes and all steps complete successfully, this is what it looks like from both sides. First, the Terraform output confirming all resources were created:

And the Azure portal showing the cluster as successfully deployed:

Step 9: Enable post-deployment outputs

Once the portal confirms the deployment is complete and Terraform finishes the apply, set deployment_completed = true in terraform.tfvars and run terraform apply once more. This allows Terraform to read the custom location output that is only available after the Azure deployment engine finishes.

Recovery Helpers

If a terraform apply times out or you lose Terraform state after a successful apply, the configuration includes two recovery variables to help you get back on track without destroying and rebuilding:

• import_deployment_settings (bool, default false): When true, imports the pre-existing deploymentSettings/default into state before the plan phase. Use this when a previous apply timed out but the resource already exists in Azure.

• import_machine_rg_role_assignment_ids (map(string), default {}): When the machine_rg_role_assign role assignments already exist in Azure and a new apply would fail with 409 Conflict, populate this map with the existing assignment GUIDs (visible in the 409 error message) and run terraform apply to import them. Reset to {} afterward.

If the Arc machine was manually deleted from Azure and you need to run terraform destroy, set enable_cluster_module = false to skip the cluster module and avoid failing Arc data-source lookups.

It is also worth noting that the cluster deployment creates additional Azure resources such as Arc extensions and logical networks that are lifecycle-managed by the cluster resource itself: they are created and destroyed together with it, so they do not need to be imported separately. The custom location is the exception, it is the only post-deployment resource that Terraform captures in state and exposes as an output, since workload modules (AVD, AKS) need to reference it. Everything else is handled by Azure as part of the cluster’s own lifecycle.

What Is Next

The cluster deployment is the foundation. The repository’s goal from the start has been to automate the full workload lifecycle on top of Azure Local, not just the cluster itself. The next steps are:

• Azure Virtual Desktop: A Terraform module for AVD host pools, session hosts and workspace configuration, deployable against the custom location created by the cluster deployment.
• AKS on Azure Local: Terraform for AKS cluster creation using the Arc-enabled Kubernetes stack, scoped to the same resource group and custom location.
• Pipelines: GitHub Actions or Azure DevOps pipeline definitions that call these Terraform configurations using the service principal created by 00_AzurePreRequisites.ps1. The goal is a single pipeline trigger that goes from a fresh Arc-registered node to a fully deployed cluster with workloads.

If you have questions, hit issues, or have already gone through a similar Terraform journey with Azure Local, I would love to hear from you in the comments below. The repository is public and pull requests are welcome.

Hello everyone, another year has gone by. As I did last year, I’ve set myself the goal of writing down a list of objectives for 2026, along with the main events that took place during 2025. This blog post is a bit more personal and less technical than usual, and its main purpose is to track the progress of my blog as well as my personal goals.

I want to be able to come back here next December and clearly see what I’ve managed to move forward and what I didn’t quite achieve, since my focus tends to change quite a bit throughout the year 😅

2025 Recap

Personal Highlights

When I look back at my photo albums, I realize just how much has happened over the year. 2025 has been a great year, at least when it comes to personal and family goals. I’ve truly enjoyed spending time with my loved ones, and I’ve gained a much healthier perspective on how I approach life.

On the professional side, I’ve also continued to grow and evolve, and you’ll see a couple of articles about that during 2026 😜. Even though I didn’t manage to achieve some of the goals I set at the beginning of last year, like attending Ignite, I did attend several conferences and surrounded myself with some truly amazing people in the professional space, not all of them coworkers, which also led to making a few new friends along the way 🤗

Professional Achievements

I wrapped up 2024 and started 2025 by earning Microsoft certifications in the areas I focused on the most throughout the year, namely Microsoft Identity and Access Administrator and Endpoint Administrator. During 2025, I was heavily focused on strengthening my overall knowledge and further developing my expertise around AVD, both in cloud and hybrid environments, with all the related challenges that come with them.

That said, this didn’t stop me from continuing to write and dedicating my free time to one of my favorite hobbies: Azure Local. In 2025, I published eight blog posts focused exclusively on Azure Local and solutions built for or around it. While I initially planned to write more about other topics, the way I approach each article, in terms of quality and the time invested, made it harder to branch out as much as I intended.

This time, I won’t include a list of articles like I did last year, nor a list of books I’m reading or have finished. This year, that list would be way too long, over 30 new books, and yes, I did finish the ones from last year as well. If anyone is curious about what I’m reading, feel free to reach out to me on LinkedIn and I’ll happily share the list 😜

Blog Stats for 2025

Overall, blog statistics have improved this year, even though I slowed down a bit in the second half in terms of posting frequency. Since this is the first year where I have a full set of statistics, it’s honestly great to see how well the blog is being received, especially considering it focuses on a very niche topic, currently almost exclusively Azure Local.

As usual, the main statistics are split across Google Analytics, Google Search Console, Clarity, Bing, LinkedIn and this year I’m also adding my GitHub Rewind.

Google Analytics

Google Search Console

Clarity

Bing Webmaster Tools

LinkedIn

GitHub

Overall, the growth this year has been quite positive, both in terms of community engagement and content. It’s true that I should have, and wanted to, publish more blog posts, but work commitments and family time took priority, which led me to put the blog slightly on hold during the second half of the year. This is clearly reflected in the statistics.

Goals for 2026

This year, I’m setting goals that are less focused on technology and the blog itself:

• Don’t start smoking again (I relapsed in 2025 😅)
• Go to the gym more regularly
• Improve my Chinese (I started studying it last year)
• Further reduce mobile phone usage and aim for less than one hour per day
• Publish one blog post per month

As always, I wrap up the year by thanking everyone who has been by my side, and especially my wife and my daughter ♥️

I wish you all a happy 2026!

Welcome to a new article on my blog. Today I’m covering a topic that helps deepen your understanding of Kubernetes: the deployment of Large Language Models (LLMs) on Azure Kubernetes Services (AKS) running on Azure Local. I’ll explain some basic concepts along the way. If anything sounds unfamiliar, I recommend checking out my previous article: Azure Local: AKS and SQL Managed Instances.

Let me also apologize up front for the slight clickbait in the title. Because of my lab setup (a laptop with 64 GB of RAM, where Azure Local only gets 34 GB) I didn’t deploy an actual LLM, but a Small Language Model (SLM). The process and implementation are identical, though.

Infrastructure Design

Before deploying anything, we should review the resources we have available and come up with a plan. As mentioned, I only have my “Azure Local Lab” running fully virtualized on my laptop. Since I can’t use a GPU on AKS here and RAM is limited (around 20 GB free for AKS, shared between the control plane and the worker nodes), I designed a simple and lightweight proof of concept.

If I had the right hardware, I would have deployed KAITO (Kubernetes AI Toolchain Operator). But since I currently don’t have a GPU available, we’ll stick to my CPU-only variant. Still, here is the KAITO architecture for reference:

To make the architecture of my lab easier to understand, I prepared a diagram based on the one shown in the official Baseline Architecture for AKS on Azure Local.

In the diagram you can see the components we’ll deploy:

• Azure Local: A single virtualized node that in turn hosts the nested AKS cluster
• AKS VMs:
  • One control plane VM that acts as API server and LoadBalancer (via MetalLB)
  • One worker node responsible for running the workloads
  • Control plane size: Standard_K8S3_v1 (4 vCPUs, 6 GB RAM)
  • Worker node size: Standard_A4_v2 (4 vCPUs, 8 GB RAM)
• Logical AKS infrastructure:
  • A dedicated namespace for LLM-related workloads
  • A single external IP from MetalLB (172.19.19.90) for Open WebUI
  • Persistent storage of 25 Gi for Ollama to keep the model files between updates

This is the workload architecture inside AKS:

The whole deployment is carried out from my laptop, which has direct Layer 2 network access to the AKS cluster. I use Az CLI, kubectl and Helm. Throughout the process I’ll point out which tool is used and why. The goal here is to test LLM capabilities in this specific environment, not to deploy a standardized production-grade AKS hybrid application.

If you’re interested in the proper deployment approach for production AKS hybrid applications, I recommend reading:
Deploy and operate apps with AKS enabled by Azure Arc on Azure Local.
That article covers pipelines, GitOps, Flux, Helm and other components, giving you a great high-level overview of the possibilities.

Another option would be a hybrid model using self-hosted agents and a Service Bearer Token (which we’ll create later in this article under “Creating a Service Account to Manage AKS Locally”). This can give you more flexibility, though I personally recommend the GitOps approach.

With the architecture clear, let’s move on to implementation.

Requirements and Tool Installation

To deploy everything, you’ll need:

• A client machine (in my case the laptop running the lab) with direct network access to the AKS cluster running on Azure Local
• The following tools installed:
  • Helm, for deploying workloads to AKS
  • Az CLI with the aksarc module, to authenticate and retrieve local AKS credentials
  • kubectl, to manage AKS directly
• An AKS cluster capable of running the workload (at least 8 GB RAM on the worker node) with temporary internet access during deployment
• The ability to add or modify local IP addresses used by the LoadBalancer (MetalLB)

Some of these requirements were already covered in the previous article, but let’s go through the ones that weren’t.

Modifying AKS

As mentioned, my lab runs nested on a laptop with 64 GB RAM, and Azure Local can only use around 36 GB or else the laptop becomes unusable. The Azure Arc Resource Bridge VM alone consumes 8 GB (or 16 GB during updates since another image is temporarily mounted). Azure Local itself uses between 4 and 8 GB depending on the moment. That leaves roughly 20 GB for AKS.

This memory has to be shared between the control plane and the worker nodes. To simplify the scenario, I limited the cluster to:

• One control plane with the minimum RAM → 6 GB
• One worker node with enough RAM to run Ollama and Open WebUI → 8 GB

I made these changes directly in the Azure Local portal under Settings > Node pools.
I also added the IP range 172.19.19.90–172.19.19.99 for MetalLB under Settings > Networking.

Once AKS is ready, we need a local management channel using a bearer token.

Creating a Service Account to Manage AKS Locally

I’ve needed bearer tokens long before AI workloads became trendy. In past projects I used them for self-hosted agents and even started a repository almost two years ago that covered this topic:
AKS Hybrid.
It’s a bit abandoned because I moved the more interesting content to other repositories.

The goal in this section is to access the AKS cluster locally. So I created this script:
12_AKSArcServiceToken.ps1

This script:

• Uses Az CLI and aksarc (installing the extension automatically if missing)
• Retrieves local AKS credentials
• Creates an admin user to manage the cluster
• Generates both a kubeconfig file and a txt file with the token
• Saves everything in the same directory
• Includes an extensive description
• Runs interactively if no parameters are provided

When the script finishes, set your kubeconfig environment variable:

$env:KUBECONFIG = "$HOME\.kube\AzSHCI\aks-arc-kube-config"

After that you can manage AKS directly with kubectl.

Now we can proceed with the workloads.

Installing Helm

We’ll use Helm to deploy the workloads. Helm is a package manager for Kubernetes. It lets you install and update complete applications using charts, which define deployments, services, volumes and configuration. In hybrid environments like AKS on Azure Local, Helm is extremely convenient because it helps you avoid writing raw YAML by hand and makes versioning and updates easier.

Install it using Winget:

winget install Helm.Helm

With Helm installed, we’re ready for the next part: deploying the LLM and exposing it on the local network.

LLM Deployment

Before going into commands, I want to explain why I chose Ollama as the base for the LLM. In a hybrid environment like AKS on Azure Local, you want something light, flexible and easy to operate. Ollama fits this perfectly. It’s one of the most popular inference engines in the community, offers a wide variety of ready-to-use models and is extremely simple to work with. It doesn’t require complex dependencies, starts quickly and lets you load models with a single command. This makes it ideal for demos, internal testing or small services running on limited hardware.

In this section we’ll deploy the full stack consisting of Ollama and Open WebUI inside the cluster. The goal is to maintain a simple, easy-to-manage architecture while still having the flexibility to run lightweight models in a controlled environment.

Below I explain each command step by step and what it means for the overall setup.

Create the namespace

kubectl create namespace slm

This creates a dedicated namespace called slm. It keeps all LLM-related components isolated, making it easier to manage, clean up or extend later.

Add Helm repositories

helm repo add otwld https://helm.otwld.com/
helm repo add open-webui https://helm.openwebui.com/
helm repo update

These are the official repositories for Ollama and Open WebUI.

Deploy Ollama as an internal service

helm install ollama otwld/ollama `
  --namespace slm `
  --set service.type=ClusterIP `
  --set ollama.port=11434 `
  --set persistentVolume.enabled=true `
  --set persistentVolume.size=20Gi `
  --set ollama.models.pull[0]=phi4-mini:latest `
  --set ollama.models.run[0]=phi4-mini:latest `
  --set resources.requests.cpu="2" `
  --set resources.requests.memory="2Gi" `
  --set resources.limits.cpu="4" `
  --set resources.limits.memory="4Gi"

Key points:

• ClusterIP keeps the service internal
• Port 11434 is the default Ollama API port
• 20 Gi persistent volume ensures models don’t have to be re-downloaded
• phi4-mini:latest is pulled and launched automatically
• Resource requests and limits prevent the node from being overloaded

Check the pods:

kubectl get pods -n slm

Deploy Open WebUI as a LoadBalancer

helm install openwebui open-webui/open-webui `
  --namespace slm `
  --set service.type=LoadBalancer `
  --set ollama.enabled=false `
  --set ollamaUrls[0]="http://ollama.slm.svc.cluster.local:11434" `
  --set persistence.enabled=true `
  --set persistence.size=5Gi

Key points:

• Exposed via MetalLB
• Connects internally to Ollama
• 5 Gi persistence for settings and local data
• You can define a static IP by adding --set loadBalancerIP=<your-ip>

Check pods again:

kubectl get pods -n slm

Final result

Retrieve the external IP:

kubectl get svc -n slm

You can now create an admin user and start using the deployed LLM:

Here are the token-per-second results. Not great, but considering everything is running triple-nested (laptop → Azure Local → AKS) with 4 vCPUs, it’s actually not bad:

With these two deployments you now have:

• A running LLM engine (Ollama) inside the cluster, only accessible internally
• A modern UI (Open WebUI) exposed via MetalLB
• Persistent storage for both models and daily usage
• A clean namespace (slm) ready for future additions

This stack is a great foundation for experimenting in a hybrid AKS environment, adding observability or even scaling to more nodes.

Conclusion

With this setup you now have a functional environment capable of running lightweight language models inside an AKS cluster on Azure Local. Helm simplifies the deployment of Ollama and Open WebUI and gives you a stable base for further experimentation. Even though this lab runs on limited resources, the approach is perfectly valid for learning more about AKS and understanding how to integrate AI workloads in a hybrid environment.

From here, you can try more models, add observability, integrate GitOps workflows or even move to solutions like KAITO if you get access to a GPU.

Welcome back to the Azure Local series. Today I want to cover a topic that comes up often with customers: observability for Azure Local. Many organizations already use on premises monitoring like Checkmk or PRTG. Integrating clusters into those tools is possible, but it usually adds moving parts such as privileged WMI from a PAW to the cluster and that can become brittle.

Microsoft ships a basic workbook under Monitoring > Insights on the Azure Local resource. It does a good job for cluster and node level views like CPU, RAM, Storage and Network. What it does not show is a deeper look at the guest VMs.

So I built Azure Local Deep Insights, a workbook that reuses the Insights pipeline and adds extra signals. The goal is to give you a single place to understand cluster health and guest VM behavior without touching the cluster from the inside. Everything runs from the Azure portal.

What the workbook looks like

The following snapshot is from Monday 27.10.2025, taken during an upgrade from 2509 to 2510. You can see how Arc Resource Bridge moved to a fresh VM during the process.

One click deployment

You can deploy the workbook with the button below. The template expects Insights to be configured on the cluster. We will adjust its Data Collection Rule in the next section.

Prefer direct code instead of the button?

• If you want the gallery template code as a .workbook file, use this link:
  https://schmitt-nieto.com/assets/img/post/2025-10-31-azure-local-deep-insights/AzureLocalDeepInsights.workbook

• If you want the raw JSON for manual deployment, use this link:
  https://schmitt-nieto.com/assets/img/post/2025-10-31-azure-local-deep-insights/AzureLocalDeepInsights.json

Prerequisites

• Azure Local resource enabled with Monitoring > Insights
• A Log Analytics Workspace dedicated to Azure Local data or shared by policy
• Permission to create or update a Data Collection Rule for the cluster

Enable Insights on the cluster

Open Monitoring > Insights, select Get started, then create:

• A Data Collection Rule
• A DCR endpoint name
• A Log Analytics Workspace

If you already use Insights you can keep your existing workspace and rule.

Once created, agents install on each node and a basic set of signals starts to flow. Initial ingestion can take about fifteen minutes. The default rule includes a few performance counters and event logs. The workbook extends this to include VM level views.

Extend the Data Collection Rule

To power the VM states and per VM performance, add the following data sources to the existing DCR. A 60 second sample rate works well in practice. It keeps costs low while keeping charts responsive.

Windows Event Logs for VM state

In your DCR open Data sources > Windows Event Logs > Custom and add both filters below.

Microsoft-Windows-Hyper-V-Worker-Admin![System[(EventID=18500 or EventID=18502 or EventID=18504 or EventID=18510 or EventID=18512 or EventID=18514 or EventID=18516 or EventID=18518 or EventID=18596 or EventID=18601)]]

Microsoft-Windows-Hyper-V-VMMS-Admin![System[(EventID=13002 or EventID=13003)]]

These events capture VM lifecycle from the Hyper-V view. The workbook uses them to build a timeline of Running, Off, Saved, Paused, Deleted, Created and more.

Performance counters for CPU

In your DCR open Data sources > Performance Counters > Custom and add the counters below to chart per VM CPU and hypervisor time.

\Hyper-V Hypervisor Virtual Processor()% Guest Run Time
\Hyper-V Hypervisor Virtual Processor()% Total Run Time
\Hyper-V Hypervisor Virtual Processor(*)% Hypervisor Run Time
\Hyper-V Hypervisor Logical Processor(_Total)% Total Run Time

Performance counters for Network

\Hyper-V Virtual Network Adapter(*)\Bytes Received/sec
\Hyper-V Virtual Network Adapter(*)\Bytes Sent/sec
\Hyper-V Virtual Switch(*)\Bytes Received/sec
\Hyper-V Virtual Switch(*)\Bytes Sent/sec

Performance counters for Memory

\Hyper-V Dynamic Memory VM(*)\Guest Available Memory
\Hyper-V Dynamic Memory VM(*)\Guest Visible Physical Memory

I plan to add more counters over time. When that happens I will update the workbook introduction and this post.

Cross check with VMFleet

If you are new to VMFleet, it is a handy PowerShell module to view cluster state from a single dashboard.

Run the following from a machine that can reach the cluster and with a user that has administrative rights.

Install-Module -Name "VMFleet"
Import-Module VMFleet
Watch-FleetCluster -Cluster <ClusterName> -Sets *

The VMFleet view is a great way to validate what you see in the workbook and get a live insights from what´s happening in the cluster.

Workbook layout

The workbook ships with the following sections:

• Overview
  Quick health and Workbook information.
• VM States
  Timeline and counts based on Hyper-V events. Filter by host and time.
• CPU
  Per VM and per host charts using guest and hypervisor run time.
• Storage
  Capacity and trends from Insights data.
• Network
  Per VM and per switch throughput.
• RAM
  Guest Available and Guest Visible memory for density planning.

Version indicator

The workbook includes a simple image based version hint. If you see Current Version in green, you are on the latest release. If you see Need Update in blue, there is a newer build with fixes or features.

Roadmap

Planned additions include AKS and AVD views that reuse the same pipeline. Priority depends on adoption, reported issues, and spare time for development.

Changelog

Azure Local brings Azure services into your data center with a consistent control plane. This page collects the docs, guides, and labs I use when planning and operating Azure Local environments. It mirrors my public GitHub list so you can search it directly on this site and keep track of updates.

GitHub source: https://github.com/schmittnieto/awesome-azure-local

Who is this for

• Cloud architects and platform engineers designing hybrid setups
• AVD administrators building or running desktops on Azure Local
• Operations teams that want proven references for day to day work

What you will find

• Official documentation and learning paths
• AVD on Azure Local
• AKS on Azure Local
• OEM notes and reference designs
• Tools, scripts, and community write ups

How to use it

• Use the table of contents to jump to a topic
• Press Ctrl F to search within the page
• Open links in a new tab to keep your place

If you spot a gap or a broken link, open an issue or pull request in the GitHub repo link.

Official

Only official links published or maintained by Microsoft or Azure.

What’s new in Azure Local (version 2604)

What’s new in hyperconverged deployments of Azure Local?

Version: 12.2604.1003.209
Release date: April 2026

A major release focused on disaggregated deployments, SAN, VM management, deployment speed and broader Azure portal improvements.

Platform and OS

• Unified operating system across all new and existing deployments.
• New baseline OS version 26100.32690, aligned with Windows Server 2025.
• Integrated System and Premier Solution hardware ship with the OS preinstalled and validated.
• Requires drivers compatible with OS 26100.32690 or Windows Server 2025.

Runtime updates

• Platform updated to .NET 8.0.26 for both .NET Runtime and ASP.NET Core.
• Also includes .NET 10.0.6 for both .NET Runtime and ASP.NET Core.

AKS and Kubernetes

• Supports Kubernetes versions 1.31.x, 1.32.x, and 1.33.x.
• Kubernetes 1.30 is no longer supported.
• KMS v2 is included and KMS v1 will be deprecated, so cluster redeployment planning is recommended.
• AKS clusters should be validated on a supported Kubernetes version before upgrading Azure Local.

Storage and deployment architecture

• Disaggregated deployments now supported, allowing Azure Local to use SAN-only storage.
• SAN support is now Generally Available (GA) and can be used alongside Storage Spaces Direct.
• Enables independent scaling of compute and storage, including clusters beyond 16 nodes.

Identity and deployment

• Local identity with Key Vault is now Generally Available (GA).
• Domain join prior to deployment is now supported.
• Update settings management is now available, giving administrators more control over how updates are applied.

Deployment and validation improvements

• Validation time reduced by up to 50% during deployment and update workflows.
• Validation can now resume from the point of failure within a three-hour window.
• Deployment time is now more consistent for clusters up to 8 nodes, with overall improvements of up to 40%.

Cluster and VM improvements

• Rack aware clustering now supports deployments using local identity with Azure Key Vault.
• GPU acceleration for Azure Local VMs is now Generally Available (GA).
• Supports attaching and detaching full GPUs (DDA) or GPU partitions (GPU-P) during VM creation or as a Day-2 operation.

Azure portal experience

• Enhanced data disk management with richer cluster-level disk views and the ability to attach existing disks directly from the VM view.
• Improved Azure Marketplace image navigation with a full-page view for image selection.
• Graceful restart is now the default for Azure Local VMs, with an option to bypass shutdown using --skip-shutdown.
• SDN management can now be enabled or disabled per network interface, using the --bypass-sdn-policies flag.

Reliability

• Includes multiple reliability improvements and general bug fixes across deployment, update, and day-to-day operations.

Azure Local

• Azure Local Product Page
  Official product page for Azure Local on the Azure website.
• What is Azure Local (Microsoft Docs)
  Introduction to Azure Local (formerly Azure Stack HCI) and its core components.
• What is new in Azure Local (Microsoft Docs)
  Lists the latest features and improvements available in Azure Local.
• Azure Local Deployment (Microsoft Docs)
  First article in a series that describes how to deploy Azure Local.
• External SAN Storage for Azure Local (Microsoft Docs)
  GA support for external Fibre Channel SAN storage in Azure Local 2604 and later, including MPIO, cluster validation, CSV integration, and Storage Path configuration.
• Azure Local Pricing
  Overview of licensing options, cost model, and subscription details.
• Azure Local Hardware Catalog
  Certified hardware from Microsoft partners.
• Azure Local Sizer
  Tool that estimates hardware requirements based on selected parameters.
• Azure Hybrid Benefit on Azure Local (Microsoft Docs)
  Link describing the contractual requirements for using Azure Hybrid Benefit with Azure Local.
• Azure Local Solutions Category
  Overview of Azure Local solution types, outlining the differences between Validated Systems, Integrated Systems, and Premier solution.
• Azure Local Security Book (Microsoft Docs)
  The Azure Local security book discusses in detail the built-in security layers found in Azure Local, from core to cloud.
• Optimize Azure Local using insights from a Well-Architected Review Assessment
  Guidance on evaluating Azure Local environments with the Well-Architected Framework assessment to identify risks, measure maturity, and improve architecture quality.
• Azure Local Baseline Architecture
  Baseline reference architecture providing workload-agnostic guidance for configuring Azure Local 2311 and later, ensuring a reliable platform for highly available virtualized and containerized workloads.
• Architecture Best Practices for Azure Local
  Guidance aligned with the Azure Well-Architected Framework that outlines architectural recommendations for Azure Local and Azure Arc, supporting hybrid and edge deployments across validated hardware.
• Rack Aware Cluster (Microsoft Docs)
  This article provides an overview of the Azure Local rack aware clustering feature, including its benefits, use cases, supported configurations, and deployment requirements. Applies only to new deployments of Azure Local.
• SDN enabled by Arc on Azure Local (Microsoft Docs)
  This article explains Software Defined Networking (SDN) enabled by Azure Arc on Azure Local. It describes SDN management methods, guidance on when to use each approach, and outlines supported and unsupported SDN scenarios.
• What’s new in Azure Local for distributed locations (Tech Community)
  Summary of Azure Ignite 2025 announcements highlighting SAN support (Preview), rack aware clustering (Preview), NVIDIA RTX PRO 6000 Blackwell Server Edition GPU support (GA), Azure Local for larger deployments (Preview), network segmentation (GA) and local identity with Azure Key Vault (Preview).
• Private Endpoints on Azure Local (Microsoft Docs)
  Overview of Azure Private Endpoints on Azure Local, including supported and unsupported scenarios, connectivity requirements, and key considerations for secure access to Azure services from on-premises environments.

AVD

• Azure Virtual Desktop documentation (Microsoft Docs)
  Complete documentation for deploying and managing Azure Virtual Desktop.
• Deploying AVD on Azure Local (Microsoft Docs)
  Step-by-step guide for running Azure Virtual Desktop in an Azure Local environment.
• AVD on Azure Local pricing
  To view pricing for Azure Virtual Desktop on Azure Local, open the “Pricing overview tab” (the rate is currently “$0.01 per virtual core per hour”).

AKS

• Architecture overview for AKS on Azure Local (Microsoft Docs)
  Architecture overview for running AKS clusters on Azure Local.
• AKS Deployment on Azure Local (Microsoft Docs)
  How to create a Kubernetes cluster in the Azure portal; clusters are Azure Arc-connected by default.
• SQL Managed Instance enabled by Azure Arc (Microsoft Docs)
  Azure SQL Managed Instance that runs on your infrastructure of choice through Arc Data Services inside an AKS cluster.
• KAITO - Kubernetes AI toolchain operator (Microsoft Docs)
  This article describes how to deploy an AI model on AKS enabled by Azure Arc with the Kubernetes AI toolchain operator (KAITO).
• Azure Kubernetes Service (AKS) Baseline Architecture for AKS on Azure Local
  Scenario describing how to design and implement a baseline architecture for Azure Kubernetes Service (AKS) running on Azure Local.
• Deploy and operate apps with AKS enabled by Azure Arc on Azure Local
  Recommendations for building an app deployment pipeline for containerized workloads using AKS enabled by Azure Arc on Azure Local, with guidance for GitOps-based operations.
• AKS on Azure Local scale requirements
  This article describes the minimum and maximum supported scale limits for AKS on Azure Local clusters and node pools.

Backup and Disaster Recovery

• Back up Azure Local virtual machines with Azure Backup Server (Microsoft Docs)
  How to back up virtual machines on Azure Local by using Microsoft Azure Backup Server.
• Azure Site Recovery on Azure Local (preview) (Microsoft Docs)
  Guide to protect Windows and Linux workloads on Azure Local during a disaster.

GitHub Repositories

• Jumpstart Local Box
  LocalBox is a turnkey solution that provides a complete sandbox for exploring Azure Local capabilities and hybrid cloud integration in a virtualized environment (hosted on Azure).
• Arc Jumpstart source code
  Source for Arc Jumpstart automation scripts and tools, supporting the official documentation site.
• Azure Local Supportability Forum
  Central repository for troubleshooting guides, known issues, and feedback used by support teams and the community.
• Azure Local sample repository
  Samples covering SDN training, security baselines, and alternate OS image download links such as the OS image tracking table.
• MSLab
  Scripts and resources to create nested Hyper-V labs, ideal for testing Azure Local in a virtual environment.

OEMs

Resources from key original equipment manufacturers with dedicated Azure Local content.

SBE

• Dell Azure Local SBE Updates
  Main entry point for Dell firmware and driver update bundles validated for Azure Local.
• Lenovo Azure Local SBE Updates
  Main entry point for Lenovo firmware and driver update bundles validated for Azure Local.
• HPE Azure Local SBE Updates
  Main entry point for HPE firmware and driver update bundles validated for Azure Local.

Dell

• Dell - Azure Local Landing Page
  Dell landing page for Azure Local.
• Dell - Deployment Guides for Azure Local
  Technical guides for Dell AX System for Azure Local or Dell AX System for Windows Server with operating system version 24H2 or Windows Server 2025 and later.
• Dell - Azure Local Hands on Labs (GitHub)
  Detailed guides on building a lab for Azure Local, tips, and deep dives.
• Dell - Dell Technologies Solutions for Microsoft Azure - Azure Local
  Website backed by the Dell GitHub repository that lists supported versions and architectural guidance for Azure Local.
• Dell - Azure Local InfoHub
  Main page that aggregates the Dell Azure Local resources mentioned above.
• Dell - SBE Page
  Page for managing firmware and driver updates on Dell hardware.
• Dell - External Storage with PowerFlex for Azure Local (White Paper)
  White paper explaining the possibility and results of adding external storage to Azure Local as Cluster Storage using Dell PowerFlex (currently the only supported option).
• Dell - How to Expand a CSV in PowerShell and Windows Admin Center
  Guide explaining how to expand a Cluster Shared Volume (CSV) or Virtual Disk in Storage Spaces Direct (S2D) using PowerShell and Windows Admin Center.

HPE

• HPE - Azure Local Landing Page
  HPE landing page for Azure Local.
• HPE - ProLiant for Azure Local Integrated Systems User Guide
  Management guide for HPE ProLiant for Azure Local, aimed at administrators who install, manage, and troubleshoot servers.
• HPE - Solutions for Azure Local - Deployment Guide
  Technical white paper with solution guidelines and example configurations for Azure Local HCI on HPE ProLiant, Alletra, and Edgeline servers.
• HPE - SBE Page
  Page for managing firmware and driver updates on HPE hardware.
• HPE - ProLiant for Azure Local Firmware and Software Compatibility Guide
  A list of components, drivers and their status for system installation, especially useful for Validated Systems.
• HPE - DL380 now part of Premier Solution category
  Announcement confirming that HPE ProLiant DL380 systems are now included in the Azure Local Premier Solution category, expanding supported configurations for enterprise and AI workloads.

Lenovo

• Lenovo - Azure Local Landing Page
  Lenovo landing page for Azure Local.
• Lenovo - ThinkAgile MX Series for Microsoft Azure Local
  Installation and deployment guides for Azure Local and related services on Lenovo hardware.
• Lenovo - SBE Page
  Page for managing firmware and driver updates on Lenovo hardware.

DataON

• DataON - Azure Local Landing Page
  DataON landing page for Azure Local.
• DataON - Azure Local Knowledge Base
  Extensive knowledge base with articles covering many Azure Local services and components.
• DataON - Azure Local Quick Reference Guide
  Set of reference guides that compare Azure Local solutions and services.
• DataON - SBE Page
  Page for managing firmware and driver updates on DataON Hardware.

Fujitsu

• Fujitsu - Azure Local Landing Page
  Fujitsu landing page for Azure Local with product information and references.
• Fujitsu - Azure Local Infographic
  Infographic that summarizes the Azure Local solution from Fujitsu.

Third Party

Independent vendors offering backup or AVD management solutions for Azure Local environments.

Backup Management

Software commonly used to perform backups of Azure Local infrastructure.

Veeam

• Veeam - Veeam Support for Azure Stack HCI
  Documentation covering backup, restore, and replication of virtual machines on the initial Azure Stack HCI by Veeam Backup & Replication.
• Veeam - VBR step by step configure Azure Stack HCI OS - Azure Local Backup
  Community article offering a deeper technical implementation of Veeam Backup for Azure Local.

Commvault

• Commvault - Azure Local Configuration and Documentation
  Guide to install and configure Commvault for protecting workloads on Azure Local.
• Commvault - Blueprint for Azure Local (PDF)
  Document explaining core concepts and Commvault capabilities for backing up Azure Local infrastructure.

AVD Management

Software used to manage Azure Virtual Desktop infrastructure on Azure Local.

Nerdio

• Nerdio - Azure Local Landing Page
  Objective overview of Azure Local, covering architecture, edge scenarios, security, connectivity, and pricing.
• Nerdio - Integrate AVD resources provisioned to Azure Local with Nerdio Manager
  Guide to integrate existing AVD infrastructure on Azure Local into Nerdio Manager for simpler management.
• Nerdio - AVD for Azure Local and Nerdio Manager
  How to configure Nerdio to manage tasks on Azure Local, including golden images and host pool configuration.

Hydra by Login VSI

• Hydra - Landing Page
  Hydra product overview from Login VSI.
• Hydra - Imaging, Rollout and Manage Azure Virtual Desktop on Azure Local
  Article by Marcel Meurer on configuring Hydra for Azure Local.

Citrix

• Citrix Virtual Desktop / App - Connecting Azure Local
  As of August 22, 2025, Citrix has published a guide on how to use Citrix Virtual Desktop and Apps with Azure Local.

Sizing

Tools to estimate Azure Local sizing and choose the right hardware.

Acuutech

• Acuutech - Azure Local Landing Page
  Objective overview of Azure Local, covering architecture, edge scenarios, security, connectivity and pricing.
• Acuutech - Scopesys Azure Local Sizing Tool
  Make the design, configuration and ordering of Azure Local or Windows Server solutions easy, by removing the technical complexity associated with scoping Azure Local or Windows Server environments.

Community

Resources that do not come from Microsoft, OEMs, or the vendors listed in the Third Party section.

Blog

To avoid an excessive list, only entire blogs are referenced, not individual posts.

• schmitt-nieto.com
  Yes, naming my own blog first is bad form, but here I share hands-on content about Azure Local implementation and management.
• hybridcore.ca
  Blog by Tiberiu Radu featuring, in my opinion, the best article on how to adopt (indirectly migrate) a VM from Hyper-V to Azure Local.
• darifer.net
  Blog by David Rivera with detailed posts in Spanish on Azure Local, covering version upgrades, deployment and comparisons with Windows Server.
• mikemdm.de
  Blog by MVP Michael Meier covering the topic Azure Local and AVD Labs.
• blog.graa.dev
  Blog by MVP Erik Grina Raassum focused on Azure Local and PowerShell implementations.
• jtpedersen.com
  Blog by MVP Jan-Tore Pedersen covering Azure Local troubleshooting and insights.
• azurelab.blog
  Italian-language blog by Luigi Pandolfino (MVP) and others about Azure Local and related topics.
• chkja.dk
  Blog by Christoffer Klarskov Jakobsen featuring Azure Local implementations.
• thisismydemo.cloud
  Blog by MVP Kristopher Turner on Azure Local and infrastructure solutions.
• francescomolfese.it
  Blog by MVP Francesco Molfese focused on Azure Local and IaC.
• auxiliumtechtalk.com
  Blog by Alyn Peden discussing real-world Azure Local scenarios.
• jakewalsh.co.uk
  Blog by MVP Jake Walsh on Azure Local implementations.
• silviodibenedetto.com
  Blog by MVP Silvio Di Benedetto on Azure Local and related topics.
• hciharrison.com
  Blog by MVP Lee Harrison covering all things Azure Local.
• kennylowe.org
  Blog by MVP Kenny Lowe on Azure Local and infrastructure.
• erniecosta.com
  Blog by MVP Ernie Costa focusing on Azure Local, Storage Spaces Direct and services.
• thomasmaurer.ch
  Blog by former Microsoft employee Thomas Maurer (Principal Program Manager & Chief Evangelist for Azure Hybrid) covering a wide range of Azure services, including Azure Local.
• gettothe.cloud
  Blog by Alex ter Neuzen focused mainly on AVD, Azure Local and IaC. I highly recommend reading it.
• azurelocal.cloud
  New website by MVP Kristopher Turner with scripts, deployments and Azure Local guidance. In my view, it is currently one of the best IaC resources in the Azure Local space.

LinkedIn

Most day-to-day news on Azure Local arrives first on LinkedIn. Here are two profiles worth following.

• Darryl van der Peijl
  One of the most active MVPs posting news and technical content. He also runs the Azure Local Insider newsletter, which I recommend.
• Dino Bordonaro
  The MVP who brought me into this space. Practical content, including the article Why Expensive Azure Local Hardware Becomes Datacenter Decoration (7 Mistakes That Turn Investment Into Inventory).
• Sven Langenfeld
  Former Microsoft Senior Azure Local Commercial Sales Specialist (DACH) who leads a large Azure Local community across technical and business topics. Founder of the LinkedIn group Azure Local Tech Talk.
• Manfred Helber
  Germany-based MVP active in community events and live streams.
• Karl Wester-Ebbinghaus
  Germany-based MVP who contributes extensively to the Azure Local Tech Talk channel.
• Flo Fox
  Former Microsoft Senior Technical Program Manager (Azure Risk) who runs the Hybrid Friends YouTube channel.

YouTube

Channels that focus primarily on Azure Local rather than hosting only an occasional video.

• Azure Hybrid Insider
  YouTube channel managed by Alexander Ortha, focused on Azure Arc, Azure Local, and hybrid integrations across Azure.
• The Hybrid Friends
  Practical use cases and deep dives in the Azure Local space.
• Manfred Helber
  German- and English-language channel by MVP Manfred Helber featuring Azure Local Show, a weekly update on Azure Local news.
• Carsten Rachfall
  German-language channel by MVP Carsten Rachfall streaming Azure Local and Azure Virtual Desktop implementations.
• I am IT Geek - Shabaz Darr
  Practical use cases by MVP Shabaz Darr covering topics like Azure Arc and Hybrid Kubernetes.

Github Repos & Tools

• schmittnieto - AzSHCI (GitHub)
  This repository brings together multiple scripts, each with its own purpose and structure, allowing you to spin up a fully functioning Azure Local environment quickly.
• bfrankMS - AzureLocal_AzStackHCI (GitHub)
  Azure Local repository with findings on automating installations of HCI, AKS, and AVD.
• Azure Local Sizing Guides (GitHub)
  Azure Local Sizing Guides provides comprehensive documentation, best practices, and tools for deploying and managing Azure services, such as AKS, AVD, and Arc-enabled SQL Managed Instance, on Azure Local. This repository is maintained by Jonathan Vella, Senior Cloud Solution Architect at Microsoft.
• VMWare to Hyper-V VM Conversion tool in Windows Admin Center (Tool)
  This agentless, cost-free tool streamlines the conversion of virtual machines from VMware to Windows Server with Hyper-V.
• S2D Capacity Calculator (Tool)
  Use this Storage Spaces Direct Calculator to estimate storage capacity, resiliency, and hardware requirements for your Storage Spaces Direct (S2D) deployment.
• Azure Local Calculator (Tool)
  A calculator currently focused mainly on pricing, since storage is covered by S2D Calculator and CPU is a separate track; created by me and available open source at Azure Local Calculator.
• AVD - FSLogix Profile Status (Tool)
  After the deprecation of FXTray, it became difficult to check the status of FSLogix profiles. With this PowerShell script, it is once again possible to verify profile status in a simple and centralized way.
• Odin for Azure Local (Tool)
  Odin, inspired by the Norse god of strategy and architecture, is an Optimal Deployment and Infrastructure Navigator for Azure Local. It provides a decision-tree interface to help select the appropriate Azure Local deployment type and instance design based on validated architecture and network configuration guidance.
• Azure Local LENS Workbook (Tool)
  A community-driven Azure Workbook that delivers deep insights into the status, compliance, and operational trends of large Azure Local fleets in minutes.
• GetToThe-Cloud - Website (GitHub)
  Source repository for the GetToThe.Cloud website, including the scripts used across the site, such as Terraform-based examples and other deployment resources.

Chats & Channels

• Azure Local Slack Channel
  Slack channel (the real Community) managed by Darryl van der Peijl that provides community support for all Azure Local questions.

Trainings

• Azure Local Accreditation 2025
  Part of the official Microsoft Learn documentation, this training path helps you get in touch with Azure Local and its core capabilities.
• Azure Local Training
  Training module covering how to deploy and register Azure Local using OEM licenses, among other key topics.

Events

Welcome to a new post! Today I’m diving into a topic I’ve discussed with countless colleagues in the Azure Virtual Desktop (AVD) community: Nerdio Scripted Actions, and in particular Windows Scripts. When I chat with fellow IT pros about how they handle AVD deployments, some lean on a variation of AVD Accelerator, others use tools like Nerdio or Hydra, and many stick with native deployment and management.

Personally, I avoid debates about which option is “best”, since each has its own niche and use cases. What I can say with confidence is that the projects where I’ve used Nerdio have been incredibly easy to adapt for clients, thanks to its simplicity. One feature that both I and my clients rely on heavily is Scripted Actions, specifically Windows Scripts, which let you automate repetitive tasks in a very straightforward way.

As more scenarios emerge that require AVD without a traditional Active Directory, using scripts to configure session hosts before they even boot up is on the rise, since applying those settings via Intune can take time. With that in mind, I created the GitHub repository “nerdio-scripted-actions,” where I’ll be hosting most of the scripts I develop. More on that later!

How Scripted Actions Work

Scripted Actions in Nerdio Manager are PowerShell scripts that run either on Windows VMs (Windows Scripts) or as Azure Runbooks. They let you customize and automate tasks, like installing software, configuring settings, or performing maintenance, at various points in the VM lifecycle.

Azure Runbooks in Nerdio

Azure Runbooks run your PowerShell scripts outside of a VM, leveraging Azure Automation. This is great for tasks that don’t require logging into each machine:

• Add data disks to VMs
• Assign or unassign users to personal desktops
• Change OS disk type of stopped VMs
• Shrink FSLogix profiles
• Enable hibernation or OS disk encryption
• Set regional settings
• Create NAT Gateways
• Power on VMs for a specific window
• Update Nerdio Manager or the AVD Agent

Windows Scripts in Nerdio

Windows Scripts run inside the VM via the Custom Script Extension. They’re perfect for installing apps, tweaking settings, or running any PowerShell code with admin rights, without interrupting user sessions.

Out of the box, Nerdio provides many ready-to-use scripts, such as:

• Installing popular apps via Chocolatey (Chrome, 7zip, VSCode, Adobe Reader…)
• Installing Microsoft 365, Teams, OneDrive
• Enabling clipboard transfer or screen-capture protection
• Applying Windows or AVD performance optimizations
• Installing FSLogix or security agents (e.g., Sophos)
• Granting local admin rights to users
• Running Windows 10/11 updates
• Restarting the AVD Agent or adjusting Sysprep settings

Where Can Scripted Actions Be Used?

Scripted Actions in Nerdio Manager can be applied at various stages of the VM lifecycle:

1. Desktop Images (Golden Images)
   • Purpose: Customize and prepare your base images.
   • Use Cases: Install or update applications, apply system optimizations, configure settings before sealing the image.
   • Application: During the “Set as image” process, you can run scripts on the source VM or the clone VM.
   • Reference: Update a Desktop Image and Hosts
2. Host Pool VM Lifecycle Events
   • Purpose: Automate tasks during VM provisioning and management.
   • Use Cases: Execute scripts when a VM is created, started, stopped, or deleted; install agents or configure settings during VM deployment.
   • Application: Assign scripts to specific VM lifecycle events within a host pool.
   • Reference: Overview of Scripted Actions
3. Re-imaging Hosts
   • Purpose: Apply updates or changes across existing session hosts.
   • Use Cases: Re-apply configurations or software installations; ensure consistency across all hosts after updating the base image.
   • Application: During the re-image process, scripts can execute to apply necessary changes.
   • Reference: Update a Desktop Image and Hosts
4. Manual Execution on Host Pools
   • Purpose: Perform ad-hoc tasks across multiple VMs.
   • Use Cases: Run maintenance scripts; apply quick fixes or updates without full redeployment.
   • Application: Use the “Run script” option within a host pool to execute scripts on selected VMs.
   • Reference: Overview of Scripted Actions
5. Scheduled Tasks
   • Purpose: Automate recurring maintenance or updates.
   • Use Cases: Schedule regular updates or clean-up tasks; automate routine maintenance scripts.
   • Application: Configure scripts to run on a defined schedule within Nerdio Manager.
   • Reference: Keeping your Image Current with Windows Updates Automatically

Scripted Action Groups in Nerdio Manager

Scripted Action Groups let you combine multiple scripted actions (Windows Scripts or Azure Runbooks) into one reusable group. These groups simplify automation by allowing a set of scripts to execute in sequence during tasks like VM provisioning or image updates.

Key Features

• Sequential Execution
  Scripts within a group run one after another in the defined order, you can rearrange them with drag & drop.

• Reusability
  Apply the same group to multiple scenarios to reduce manual effort and ensure consistency.

• Tag Support
  Assign tags to groups for better organization and filtering.

Limitations

• Maximum of 20 Scripts per Group
• No Nesting: You cannot include one group inside another.
• Flat Execution: The system treats each script individually; order matters.

My Scripted Actions Repository

To help you get started with Nerdio Scripted Actions, I’ve curated a dedicated GitHub repo at nerdio-scripted-actions. Inside, you’ll find a growing collection of PowerShell scripts designed for everything from image customization to day-to-day host-pool tasks.

Nerdio maintains its own library under NMW Scripted Actions, many of which are drawn from Microsoft’s RDS-Templates project. To illustrate, here are two complementary scripts:

• Install language packs.ps1 (Nerdio): Install language packs.ps1
• InstallLanguagePacks.ps1 (RDS-Templates): InstallLanguagePacks.ps1

The Nerdio script begins with a standardized header that defines metadata and user-input variables. For example:

<#
  Author: Akash Chawla
  Source: https://github.com/Azure/RDS-Templates/tree/master/CustomImageTemplateScripts/CustomImageTemplateScripts_2024-03-27
#>

#description: Install language packs
#execution mode: Individual
#tags: Microsoft, Custom Image Template Scripts
<#variables:
{
  "LanguageList": {
    "Description": "Select any additional languages to be added",
    "DisplayName": "Languages"
  }
}
#>

In the variables section, you define the fields that end users will interact with, here, a simple language picker:

By hosting these scripts in your own repo, you gain full control over versioning, customizations and integration with Nerdio’s “Script repositories” feature, making your AVD automation both transparent and repeatable.

Integrating the Repository in Nerdio

1. Go to Settings > Environment > Integrations > Script repositories:

   

2. Add the repository URL and select folders to sync:

   

3. Verify the scripts under Scripted Actions > Windows Scripts > Filter By Source:

   

My Scripts

Currently, the repository shows three scripts, with three more in development. Here’s what you’ll find today:

• Hide or Unhide Drives in File Explorer
  Hide one or more drives (e.g., “C,D,E”) from File Explorer to simplify the UI. Users can still access them, it just cleans up the view. You can also revert the change without re-entering drive letters.

• Disable Office Updates Task
  Prevent SDXHelper.exe from running in multi-user environments. This script disables, or re-enables, the Office update task on session hosts.

• OneDrive Remote App Configuration
  Apply the settings required to run OneDrive in AVD Remote App pools (Microsoft documentation). This script also supports reverting the changes.

Running My Scripts Independently of Nerdio

If you need to test or apply these scripts on a Windows VM without connecting to Nerdio Manager, you can execute them directly. This method is handy for quick troubleshooting or one-off tasks:

1. Download the desired .ps1 file from the GitHub repo to your local machine.
2. Open PowerShell as an Administrator.
3. Bypass the execution policy and run the script. For example, to hide drives “C” and “D”:

Set-ExecutionPolicy Bypass -Scope Process -Force
.\HideDrives.ps1 -Action Hide -DrivesToHide "C,D"

That’s it, your script will run immediately, just as it would within Nerdio’s framework.

How to Collaborate

I’ll keep this repo updated as new requests and use cases come up. I’m also working on integrating my scripts into the official Nerdio repository via this pull request. If you want to contribute:

• Open an Issue on the repo: https://github.com/schmittnieto/nerdio-scripted-actions/issues
• Fork the repo and submit a Pull Request with your scripts
• Not comfortable on GitHub? Feel free to reach out to me on LinkedIn and we can collaborate privately.

Conclusion

Nerdio Scripted Actions, especially Windows Scripts, can significantly streamline your AVD deployments by automating everything from app installation to configuration tweaks. Whether you’re customizing golden images, automating host pool events, or running ad-hoc maintenance, these scripts give you the flexibility and control you need. I hope you find my repository helpful, and I’m eager to see what you build with it. Feel free to dive in, suggest enhancements, or share your own scripts!

Links Used in This Article

Welcome to a new article in the Chronicloud Series. In this post, I’ll be addressing a topic that is always a key consideration when using Azure Local: Backup and Disaster Recovery.

If you’re accustomed to using infrastructure in Azure and haven’t yet experienced Azure Local, this subject might catch you off guard, since the backup and restore process here is completely different. Azure Local is deployed in your chosen location, whether that’s a data center, your company’s basement, or, in my case, my laptop, and it follows backup and recovery procedures that are almost identical to those used in Onpremises environments like Windows Server 2025, VMware, Nutanix, or similar systems.

This means that, for now, you need an on-premises backup infrastructure to secure your workloads. There is one small exception: Azure Site Recovery, which is currently in Preview and will be discussed later in this article.

Throughout this post, I will explore various backup solutions and possibilities, detailing both their dependencies/requirements and their limitations. Although I won’t implement a full technical backup solution (since my testing infrastructure is too limited for such a setup and would take too long to produce a satisfactory result), I will cover Azure Site Recovery in detail as I have a few specific insights to share.

Additionally, I’ll delve into the topic of hydration, a subject I’ve discussed in previous articles, examining its challenges and current possibilities. In my view, this aspect plays a crucial role in the VM recovery process.

The Challenge: Hydration and VM Recovery

For those who aren’t familiar with what I mean by Hydration, I’ll try to explain it simply. Hydration is a process where an Azure Local VM, managed and configured as if it were a Hyper-V VM (e.g. using PowerShell commands, Hyper-V Manager, Cluster Manager, or Windows Admin Center), is transformed into an ARC VM. In other words, it converts a “local” VM into a “hybrid” VM that can be managed from the Azure portal.

Regarding the recovery process and hydration of a VM in Azure Local, here’s what I can say:

• If your goal is solely to cover the backup and recovery of VMs within a single Azure Local Cluster, I can confidently state that this is currently achievable through various solutions. Restoring the VM in the cluster returns it to its original state, meaning no hydration is needed because the VM functions normally.
• Conversely, if your objective is to back up and restore across different Azure Local Clusters, there are a couple of limitations and challenges to consider. The primary challenge is restoring an ARC VM in an Azure Local Cluster different from the original.
  • Since there isn’t currently a hydration service in Azure Local that allows registering (or re-registering) the VM in a cluster other than the one it was originally set up on, the only thing that gets restored is the VM as a local resource (with its CPU, RAM, storage configuration, etc.).
  • This doesn’t mean the VM can’t be used in the new cluster; it will simply operate as a basic Hyper-V VM and won’t have the capabilities of an ARC VM since it can’t utilize the ARC Resource Bridge it was configured with.

As for when and how a hydration service will be implemented in Azure Local, there’s no current information available about its roadmap or status. Despite the absence of a hydration service, there are a couple of workarounds that can help you implement a local VM as an ARC VM, which I’ll explore in more detail in the next section.

Workarounds for Hydration

Personally, I only know of one “functional” workaround and one “theoretical” workaround. Neither is supported or documented by Microsoft; what I’m about to describe isn’t designed or validated by Microsoft, this is simply based on my experience as a provisional measure until the service becomes available.

Let’s start with the functional workaround. This method uses Azure Migrate (in preview) and requires a server running Hyper-V to migrate the VM to Azure Local. The migration process, which I’ll detail in my next blog post (Azure Local: Planning, Sizing & Migration), is as follows:

This process allows us to migrate the VM from a Hyper-V environment to Azure Local and have it appear as an ARC VM by registering it with the ARC Resource Bridge. However, if the VM we want to “migrate” is actually a restoration of a VM from another Azure Local cluster, a couple of preparatory steps are necessary to ensure it registers correctly in the new cluster. The most important step noted in the documentation is:

• Verify that none of the VMs you plan to migrate have the Azure Connected Machine Agent installed. For more information, see FAQ.

In other words, you need to uninstall the “Azure Connected Machine Agent” and, to avoid issues, also remove any existing resource in the Azure Portal previously linked to that VM. There are also certain limitations with this process, such as the MAC address of the network interfaces changing (link). This usually isn’t a problem, but it wouldn’t be the first time that an application requires a specific MAC address due to licensing concerns 🥲.

That said, since my goal isn’t to describe in detail how Azure Migrate works (I’ll cover that in my next article), for those eager to understand the process, I recommend a great article by Kenny Lowe which details the migration of a local VM (located on a cluster node) to an ARC VM within the same Azure Local cluster, in other words, performing the hydration process via Azure Migrate: Migrating VMs to Azure Stack HCI 23H2.

Once the functional workaround has been discussed, we move on to the theoretical workaround, which I haven’t yet tested and have serious doubts about. The theoretical workaround is based on some responses from users in the following post, which states:

• You can deploy a new Azure Stack HCI virtual machine in your newly redeployed cluster and then overwrite the existing virtual disk with your backup data.

I personally believe this might work only if the VM wasn’t previously registered with ARC and assuming that the ARC Resource Bridge would initiate a new onboarding process upon detecting that the VM isn’t “configured correctly” (which it doesn’t). However, as mentioned, this is an approach I haven’t been able to verify, and I remain quite skeptical about its viability when restoring an ARC VM from another Azure Local cluster.

I plan to take some time in the coming weeks to validate the theoretical workaround and, if it proves functional, update this article to reflect my experiences. For now, though, we’ve covered enough on hydration in Azure Local, so let’s move on to the next topic, which should be the main focus of this article: Backup.

Backup

This is a topic that always gives us headaches but is fundamental to any infrastructure, whether due to hardware issues, physical incidents (floods, fires…), configuration mistakes (deleted files, corrupt VMs…) or security breaches (ransomware). Without a robust backup, you’re really in trouble…

I won’t delve too deeply into the basic concepts of backup and recovery here, as that would make this article excessively long. Instead, I encourage you to explore these fundamentals through a few links from one of the most popular solutions my clients use (Veeam):

• What is the 3-2-1 backup rule?
• RPO and RTO: What’s the Difference?
• Best Practices: Assess, Design, Build, Operate & Secure
• Backup resource calculator
• Backup servers and repositories should not depend on the infrastructure (e.g. not be part of AD, be on a separate physical server…)

As I mentioned in the introduction, even though Azure Local is part of Azure’s hybrid infrastructure, it does not include the native Azure Backup service. On reflection, this makes sense, as it would be very difficult, costly, and inefficient to maintain the required RPOs and RTOs by continuously backing up workloads solely over the Internet (WAN).

There is an approach that comes close to this, which we’ll cover later, called Azure Site Recovery (ASR). This means that if you design an Azure Local solution, you also need to design or adapt your backup strategy. In the sections that follow, I’ll discuss a couple of solutions for the various workloads that can be implemented in Azure Local, along with their dependencies, advantages, and benefits.

Host

Although, in theory, you could back up the hosts (while, of course, avoiding the Cluster Shared Volume), I do not recommend this practice since it is not one of Microsoft’s recommended best practices. In case one of the nodes suffers hardware damage, there is some excellent documentation that outlines the node repair process:

That said, this guide does not apply to single-node servers. This is the only case where, besides backing up the workloads, I would recommend backing up the node itself (without needing to back up C:\ClusterStorage, as it is redundant). A failure here would constitute a “Disaster Recovery” scenario, and if we want to maintain the ARC VMs and the entire configured infrastructure, it could be a possible method to achieve that.

While I personally wouldn’t back up the entire nodes of a cluster, there are certain folders on each node that I like to keep copies of, and these are:

• C:\ProgramData\AzureConnectedMachineAgent\Log
• C:\MASLogs
• C:\Agents
• C:\CloudContent
• C:\CloudDeployment
• C:\ClusterStorage\Infrastructure_1\ArcHci

The last folder ideally should not be backed up, but since it contains the log of the last time the ARC Resource Bridge was provisioned in the event of Disaster Recovery and a cluster rebuild, it may contain relevant information.

VMs

Since VMs are the primary workload in Azure Local (at least in my experience), there are several options for creating backups. Of the solutions I’ll present below, I want to highlight that they are just two of the seven currently available for Azure Local, in other words, when it comes to backing up VMs, you have plenty of choices.

Before diving into the two solutions I will cover, please note that depending on the workload you intend to run, it makes more or less sense to perform backups (as well as to determine the backup frequency, i.e., how many backups of the VM are performed each day). A couple of examples of this would be:

• AVD Multiuser Session Host: Personally, since I usually manage session hosts using Golden Images, I don’t require a backup of the VMs because the valuable data (user profiles and/or data directories) is already secured via the File Server backup.
• SQL Server: SQL servers, due to the high volume of changes throughout the day, typically have a much more intensive backup configuration compared to other VMs. Common practices in mission-critical servers include backing up transaction logs every 30 minutes, performing incremental backups every hour, and a full backup once a day.

All the solutions we will analyze (and indeed all those available) share the same problem when restoring VMs in an ALR (Alternate Location Recovery), that is, in another cluster: the VMs are restored as Hyper-V VMs rather than as ARC VMs, which is something we discussed in the Hydration section.

Microsoft Azure Backup Server

Before anything, I want to share the official guide on how to configure Azure Local Backup with Azure Backup Server, which also outlines the supported scenarios.

In my experience, MABS is not one of the most widespread solutions in the market (at least not in Germany 🤔), likely due to its limitations and performance compared to other infrastructure platforms (aside from Azure Local and VMware) and, conversely, the services offered by other solutions. However, it can be a more economically efficient option, as it does not incur additional licensing costs for the backup service, although it should be taken into account that for large data volumes (many terabytes), the cost of storing information in Azure Backup Vaults can be higher compared to other alternatives..

Regarding integration with Azure, MABS integrates natively with hybrid services (storing data in an Azure Backup Vault), which, in my view, simplifies and eases its management.

When it comes to backing up VMs, MABS offers the following features:

• Native Hyper-V/Azure Local support: Backs up VMs in an Azure Local cluster with an agent on each node.
• Host-level backup: Protects VMs even when stored on local storage, DAS, or CSV (Cluster Shared Volumes).
• Cluster mobility: Detects live migration; if a VM moves to a different node, MABS continues protecting it on the new host without manual intervention.
• Limitation: It does not support moving a VM to a different cluster (only within the same cluster).

Specific workload limitations/characteristics include:

• Domain Controller: It does not offer granular restoration of individual AD objects from the MABS console (there is no “item level restore” for AD); you would need to recover a System State backup and use AD tools (or the AD Recycle Bin) to restore specific objects.
• SQL Server: SQL backups can be sent to an Azure Recovery Services Vault for long-term retention (up to 99 years), with storage isolated from the local infrastructure. It does not protect databases whose files reside on remote shared resources (SMB), nor certain complex scenarios of Availability Groups with mismatched names.

For the proper operation and backup of the aforementioned specific workloads, Guest Agents are required, which slightly increases the complexity of the solution.

Concurrency and parallel processing: Historically, MABS had restrictions on the number of simultaneous tasks, but in version 4 it supports up to 8 parallel backups/restorations of VMs by default (configurable via settings).

Storage efficiency: MABS can leverage Windows Server deduplication on its backup volumes to reduce consumed space (a strategy inherited from System Center DPM). Additionally, it compresses and encrypts the data sent to the Azure Backup Vault.

Load on the protected systems: MABS, by using agents on each server/VM for certain tasks (for example, agents within VMs for SQL or for the System State of Domain Controllers), can impose CPU/memory usage on the guest systems during the backup window (due to VSS operations, encryption, and data transfer).

File restoration: It allows restoring individual files or folders to their original or an alternate location. MABS v4 introduced the ability to browse recovery points in Azure without fully downloading them, making it easier to restore specific files from the cloud (link).

Data encryption at rest: All content backed up in the Recovery Services Vault (in Azure) is encrypted by default with 256-bit AES (link). In on-premise MABS, data stored locally on the backup server can be encrypted using BitLocker or the operating system’s EFS if needed, but there is no native encryption option in MABS for that local storage beyond what Windows provides.

Backup immutability (protection against deletion/modification): On-premises, MABS does not have an immutability mechanism for its backup disks beyond protecting server access. However, if you use immutability in the Recovery Services Vault and “lock” it, Azure guarantees that no backup can be deleted or altered before its expiration date according to the configured policy (link). This adds a layer of protection against accidental or malicious deletions, even an attacker with access would not be able to delete the current backups.

Given all of this, if you’re starting from a “Green Field approach” and do not have highly specific backup requirements for your workloads, MABS is a solution that maybe fits your scenario. It is a Microsoft solution for a Microsoft product, and despite not having all the features that other backup providers might offer, I believe the features it does provide are more than sufficient to meet your needs.

Veeam

Why Veeam? In many of the infrastructures where I’ve implemented Azure Local or Windows Server, a backup solution was already in place, and in my experience, Veeam was the most widely adopted by my clients. I must say it offers certain benefits over MABS in terms of backup and recovery. Like every backup solution for Azure Local, it also encounters the hydration issue, meaning that its use should be evaluated based on its local infrastructure features rather than its Azure integration.

Without getting too lengthy, I’ll summarize the advantages Veeam brings compared to MABS, as well as the supported scenarios and requirements for integrating Veeam in Azure Local. I’ll start with the integration requirements, which are brief and specific:

• Integration with Windows Defender Application Control (WDAC) in Azure Local: One of the main issues I’ve seen in the past is that Veeam couldn’t be integrated into Azure Local because WDAC blocked its use. Using this article, you can add a supplemental policy to allow Veeam to integrate.
• Requirements and support: The supported scenarios and requirements are detailed in this article. They can be summarized as:
  • Minimum version of Veeam Backup & Replication for Azure Local (version 25398.X): The current minimum supported version is 12.1 (build 12.1.0.2131).
  • No support for Azure Arc VM management (Hydration): Azure Arc VMs in a “Running” state can be backed up. Upon restore, these VMs become standard Hyper-V workloads. If a VM is restored (and the original is no longer present) within the period that Azure Arc can reconnect (45 days), the Azure Arc connection should persist if restored to the same cluster.

I also want to mention an issue that may occur in Azure Local Clusters running an older 22H2 version that hasn’t been updated regularly. This is the “Hyper-V Resilient Change Tracking Performance Issues”, where general Hyper-V OS performance degradation can occur when using a backup solution to export Hyper-V VM snapshots during backup operations. The fix involves installing the Cumulative Update from February 11 and adding the following registry value:

• Key Location: HKLM\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides
• Value Name: 636159629
• Value Type: DWORD (32-Bit) Value
• Value Data: 1

Now, let’s examine the supported scenarios and the advantages of Veeam over MABS for backing up on-premises infrastructure:

• Support for Hyper-V/Azure Local: Veeam backs up VMs in local clusters without needing to install agents on each machine, leveraging hypervisor-level integration.
• Host-level backup: It protects VMs even on local storage, DAS, or CSV (Cluster Shared Volumes), ensuring continuity in environments with distributed storage.
• Cluster mobility: Veeam automatically detects VM movements (Live Migration) and continues applying backup policies without manual intervention.
• Additional functionality: It includes Instant VM Recovery, which allows you to boot a virtual machine directly from the backup, significantly reducing restoration times.
• Support for additional platforms: Veeam supports backups for Proxmox, Nutanix, Oracle, and Red Hat in addition to Azure Local and VMware (link).

• Live Migration: Although VMs won’t be rehydrated in the new cluster, locally (as Hyper-V VMs) it is possible to move VMs between clusters while retaining the backup. In fact, you can even move VMs from other platforms (e.g., VMware) via Instant Recovery. That said, I personally do not recommend this approach if the goal is to migrate to Azure Local, as it would forfeit the possibility of integrating the VM as an Arc VM, a process that can be achieved with Azure Migrate, which I’ll cover in my next blog post. Here’s a schematic of the process:

• Workload-specific features:
  • Domain Controller: Veeam protects domain controllers through its integration with VSS, ensuring consistent backups. With Veeam Explorer for Active Directory, you can perform granular restores of individual objects, making it easy to recover specific users or groups without restoring the entire system.
  • SQL Server: Veeam natively backs up SQL databases using Application-Aware Processing, ensuring data consistency during backups. It also allows sending backups to cloud repositories (e.g., Azure Blob Storage) for long-term retention. In complex scenarios (like Always On Availability Groups or databases with files on remote shared resources), specific configuration adjustments may be necessary for complete protection.
• Concurrency and parallel processing: Veeam is designed for high-demand environments, enabling multiple backup and restore tasks to run in parallel. Thanks to its distributed architecture, with scalable backup proxies and repositories, it efficiently manages large volumes of data in line with the available infrastructure capacity.
• Storage efficiency: Veeam applies deduplication and compression techniques at the job level, significantly reducing backup storage space. It also offers the option to integrate external deduplication appliances, further optimizing storage usage, especially in environments with high data volumes or frequent changes.
• Data encryption at rest: Veeam natively encrypts backup files using AES 256-bit (link), ensuring that stored information, whether in local repositories or in the cloud, remains protected. This option is set at the job or repository level, providing added security against unauthorized access.
• Backup immutability (protection against deletion/modification): Recent versions of Veeam offer options to configure immutable repositories to protect against unauthorized deletions or modifications. This is achievable both in local repositories (using Linux systems with immutable attributes) and in cloud storage environments, leveraging the immutability features of Azure Blob Storage.

Not everything is an advantage, so here are additional factors to consider when deciding on a backup infrastructure:

• No use of Azure Backup Vaults: Although you can configure Azure Blob Storage as the backup storage, Veeam does not use vaults natively. This means the solution isn’t integrated with Azure out-of-the-box and requires an additional management portal to handle or verify backups. Personally, I’m a fan of the configurations possible with Veeam Backup Monitoring & Replication Alarms, so I don’t see this as a major limitation.
• Know-How: While Veeam is user-friendly, integrating it into a new infrastructure requires specific prior knowledge, a topic that could easily fill 10 blog posts 😅. Veeam backs up infrastructures more effectively and efficiently, but there is an initial learning curve. They even offer a specialized certification program for experts that covers these topics in depth (link).
• Licensing Costs: I won’t commit to a specific number as costs vary per infrastructure, but you can calculate expenses using their pricing calculator. Compared to MABS, depending on the amount of data to back up, Veeam can be a much more expensive option.
• Additional Infrastructure: Although Veeam can be configured with an infrastructure similar to MABS, leveraging its full potential might require designing additional infrastructure, which can incur extra monetary and operational costs.

Taking all these points into account, I conclude that Veeam is a more advanced, focused, and specialized backup solution that offers many more options than MABS, though it does come with additional costs that must be planned for in advance. In my experience, especially in the German market, if a client has on-premises infrastructure and a backup system, it is very likely to be Veeam.

Trusted launch VMs

After covering the VMs workload in the previous section, a new question arises: how are Trusted launch VMs backed up and restored? The main difference compared to standard Azure Local VMs is the following:
Trusted Launch VMs in Azure Local environments differ from standard VMs because they use a dedicated key to secure the state of the VM guest, including the virtual TPM state when it is not active. This key is kept safe in a local key vault located on the same Azure Local system as the VM. Additionally, the guest state of a Trusted Launch VM is saved in two separate files: one file contains the guest state and the other contains the runtime state. As a result, any backup or restoration process for a Trusted Launch VM must include all VM files along with the protection key.

I want to highlight the following message from Microsoft’s documentation regarding Backup and Restore of Trusted launch VMs:

Backup and disaster recovery tooling support: Currently, Trusted launch for Azure Local VMs do not support any third-party or Microsoft-owned backup and disaster recovery tools, including but not limited to, Azure Backup, Azure Site Recovery, Veeam, and Commvault. If there arises a need to move a Trusted launch for Azure Local TVM to an alternate cluster, see the manual process Manual backup and recovery of Trusted launch for Azure Local VMs to manage all the necessary files and VM protection key to ensure that the VM can be successfully restored.

Because of this, before deploying a Trusted Launch VM, you need to consider how to back it up, as the current solution is neither elegant nor integrated into standard backup solutions. My main advice is that if you require Trusted Launch VMs, make sure to set up an automation (for example, a PowerShell script that saves the results of the manual process into a CSV which is then backed up) to ensure they are properly secured. Otherwise, you might end up with backups that cannot be used, and as they say, a backup that cannot be restored is as good as having no backup at all (even worse, since you’re wasting space and resources 😅).

With that in mind, the manual backup and restore process is detailed perfectly in the following summary.

Manual backup and recovery of Trusted launch VMs

Overview
Trusted Launch Arc VMs use a VM Guest State Protection (GSP) key stored in a local key vault to secure the VM guest state (including the vTPM state). The guest state is divided into two files: VM Guest State (VMGS) and VM Runtime State (VMRS). Losing the GSP key prevents the VM from booting, so it’s essential to back up both the VM files and the GSP key regularly.

Manual Backup Process

1. Export the VM Files
   Use Export-VM to back up all VM files (VMGS and VMRS).

2. Back Up the GSP Key

   • On the Azure Local system with the backup key vault:
     Create a wrapping key and download its PEM file by running:

     New-MocKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type RSA -size 2048
     

     Get-MocKeyPublicKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -outputFile wrappingKey.pem
     

   • On the Azure Local system hosting the VM:
     Copy the PEM file, verify the VM owner node, and retrieve the VM ID:

     Get-ClusterGroup <VM name>
     

     (Get-VM -Name <VM name>).vmid
     

     Then, export the GSP key:

     Export-MocKey -name <VM ID> -wrappingKeyName wrappingKey -wrappingPubKeyFile wrappingKey.pem -outFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -size 256
     

   • Return to the backup key vault system:
     Transfer the <VM ID> and <VM ID>.json files, and import the GSP key:

     Import-MocKey -name <VM ID> -importKeyFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type AES -size 256
     

Manual Restore Process

1. Prepare the Target System
   On the Azure Local system where you want to restore the VM, create a new wrapping key and download its PEM file:

     New-MocKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type RSA -size 2048
   

     Get-MocKeyPublicKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -outputFile wrappingKey.pem
   

2. Export the GSP Key from the Backup
   On the system with the backup key vault, copy the PEM file to the target system. Then, obtain the VM ID (from the VM’s config file or by executing):

     (Get-VM -Name <VM name>).vmid
   

   Export the GSP key using:

     Export-MocKey -name <VM ID> -wrappingKeyName wrappingKey -wrappingPubKeyFile wrappingKey.pem -outFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -size 256
   

3. Import the GSP Key on the Target System
   On the target system, copy the <VM ID> and <VM ID>.json files and import the GSP key:

     Import-MocKey -name <VM ID> -importKeyFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type AES -size 256
   

Important:
Ensure that the GSP key is restored on the target system before starting the VM. If the VM is started without the restored key, the system will generate a new GSP key, which may cause errors. In such a case, remove the incorrect key with:

Remove-MocKey -name <vm id> -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault

Cluster Storage

At this point, let’s focus on Cluster Shared Volumes (CSV). Normally, additional backups of the storage aren’t required if the workloads have already been backed up. However, as I mentioned in the Host Backup section, in certain scenarios it might be useful to back up specific CSV folders. I primarily focus on the Infrastructure folders (especially their logs), and if we use Golden Images for deploying certain workloads that are stored on the CSV, you might also want to back those up.

Another possibility, although I don’t recommend implementing it directly, is that if your cluster is designed solely for AVD usage, you could use one of the CSVs as the storage directory for FSLogix user profiles. I wouldn’t recommend this approach directly because it implies that the cluster is in the same Active Directory as the AVD users, and in turn, it would expose a shared folder within a CSV where users can store their profiles. I am in favor of using Fabric Domains (as I mentioned in the previous article and will discuss further in the next post) that are separate from the main domain. However, I understand that if the only workload is AVD, this implementation might be considered to minimize costs and leverage the storage space and performance of the cluster.

I don’t want to dwell too much on this type of backup, since all backup solutions can cover this function. As noted throughout this post, such a backup is only necessary if CSVs are used for specific workloads, in which case I assume the designer will inform the person responsible for backups to ensure they are properly secured.

Key Vault

The Key Vault required for installing Azure Local is not a typical workload. However, considering that the Azure Local deployment with local identities without the need for Active Directory (currently in preview) makes full use of the Key Vault for cluster operations, it is important to note that this component should be backed up in the future.

Currently, and likely in your case as well, the Key Vault contains only three secrets: the AzureStackLCMUserCredential, the DefaultARBApplication, and the LocalAdminCredential. These are created during installation and are valid for one year from deployment. As mentioned, these are not critical because, in case of errors (for example, if the BitLocker key is needed), they can be resolved using other methods. However, given the upcoming changes, I personally recommend taking a look at this article: Recover a deleted key vault and resume backup.

The Key Vault backup process is detailed in the following section.

Key Vault Backup Process

Overview
This article outlines how to back up individual secrets, keys, and certificates from your Azure Key Vault. The backup creates an offline, encrypted snapshot of each object to protect against loss of access to your key vault.

Key Considerations

• Backup Scope: Each key vault object must be backed up individually; there isn’t an option to back up the entire vault in one operation.
• Version Limits: Only up to 500 previous versions per object can be backed up. Attempting to back up objects beyond this limit may result in errors.
• Snapshot Nature: The backup is a point-in-time encrypted blob. Because the blob cannot be decrypted outside of Azure, it must be restored within the same Azure subscription and geographic region.
• Operational Challenges: Managing multiple backups, logs, and permissions can be complex, especially as secrets rotate or expire.

Prerequisites

• You need Contributor-level (or higher) permissions on your Azure subscription.
• A primary key vault containing the objects to back up and a secondary key vault where these objects can later be restored are required.

Backup and Restore Using the Azure Portal

• Backup:
  1. Go to your key vault in the Azure portal.
  2. Select the desired object (secret, key, or certificate) and click Download Backup.
  3. Download and securely store the encrypted blob.
• Restore:
  1. In your key vault, select the object type and click Restore Backup.
  2. Locate the encrypted blob and confirm to complete the restoration.

Backup and Restore Using Azure PowerShell

1. Log in and Set Context:

   Connect-AzAccount
   Set-AzContext -Subscription '{AZURE SUBSCRIPTION ID}'
   

2. Backup Commands:
   • Back up a certificate:

     Backup-AzKeyVaultCertificate -VaultName '{Key Vault Name}' -Name '{Certificate Name}'
     

   • Back up a key:

     Backup-AzKeyVaultKey -VaultName '{Key Vault Name}' -Name '{Key Name}'
     

   • Back up a secret:

     Backup-AzKeyVaultSecret -VaultName '{Key Vault Name}' -Name '{Secret Name}'
     

3. Restore Commands:
   • Restore a certificate:

     Restore-AzKeyVaultCertificate -VaultName '{Key Vault Name}' -InputFile '{File Path}'
     

   • Restore a key:

     Restore-AzKeyVaultKey -VaultName '{Key Vault Name}' -InputFile '{File Path}'
     

   • Restore a secret:

     Restore-AzKeyVaultSecret -VaultName '{Key Vault Name}' -InputFile '{File Path}'
     

AKS

We are now nearing the end of the workloads to be backed up, and of course, we cannot forget AKS Hybrid. Many may wonder why one should back up AKS when, in theory, simply backing up the repository from which AKS is provisioned would suffice. Well, yes and no. I consider the repository as a golden image; if AKS is used in a generalized manner without the need for permanent storage and if the workload itself generates the backup (through copies to external servers or by using backed-up services), then you might not require a backup for AKS since the backup of your repository is more than enough. However, if you are using permanent storage and/or services that require a backup (for example, MongoDB) that are not handled at the application level, then a dedicated backup solution is necessary.

To date, it is not possible to perform native backups of AKS Hybrid in Azure Local, unlike in Azure Public where AKS has its integrated backup service. Therefore, external solutions are required to carry out this process. The only solution I have considered so far is Velero, mainly because there is an article in the documentation (Back up, restore workload clusters using Velero) that explains how to do it, although it should be noted that this article applies to AKS on Azure Local 22H2, AKS on Windows Server. Since I have not yet integrated Velero in Azure Local 23H2, I cannot say what limitations it may present in the new version of Azure Loca (related to Arc Resource Bridge). Perhaps in the coming weeks, with a bit more time, I will take a closer look at this matter. That said, if your applications have been configured correctly, you likely won’t need to back them up.

There is an excellent guide written by Alper Can that details the entire backup and restore process in Azure Local using a local backup via Minio Storage installed on an Ubuntu VM. For more information, you can check out the article here.

SQL Managed Instances enabled by Azure Arc

Although this workload should technically fall under AKS, since it involves Arc Dataservices running within AKS, there is currently no native solution for backing up this workload (or at least I haven’t found adequate information on this topic so far). Therefore, I have decided to dedicate a separate section to it.
Because it is nearly impossible to back up AKS, the Data Controller within it (with its custom location), and then the SQL Managed Instance enabled by Azure Arc, an ordeal in terms of recovery, I have concluded that the most effective, efficient, and straightforward method is to perform a backup using SQL Server Management Studio.
Perhaps BCP (Bulk Copy Program) could be used to automate the process, but I personally have no experience with it. From what little I have researched, there are some important considerations:

• This method transfers data only. Schema objects such as stored procedures, triggers, and indexes are not transferred and must be handled separately.
• Ensure that the target tables exist in the destination database before importing.

Once I have a larger lab where I can run SQL Managed Instances, I will try to find a better solution for this issue (and I will also test Velero). For now, however, you will have to settle with SQL Server Management Studio 😅.

Limitations

Based on the only article I have found on this subject in the documentation (Limitations of SQL Managed Instance enabled by Azure Arc), here are the limitations of backup in SQL Managed Instances enabled by Azure Arc. Note that I have not been able to find more information on “Automated Backup,” which is why I still consider that the most efficient way to perform the backup is through SQL Server Management Studio.

Backup and Restore Limitations

• Automated Backups:
  • User databases using the SIMPLE recovery model aren’t included in backups.
  • The system database model is excluded to avoid interfering with administrative operations like creating or deleting databases, which may lock the database during such tasks.
• Point-in-Time Restore (PITR):
  • Restoring a database is confined to the same Arc-enabled SQL Managed Instance where the backup was created; cross-instance restores are not supported.
  • Renaming databases during a point-in-time restore isn’t supported.
  • There is currently no support for restoring databases that have Transparent Data Encryption (TDE) enabled.
  • Once a database is deleted, it cannot be restored.

Other Limitations

• Transactional replication is not available.
• Log shipping is blocked.
• Every user database must operate under a full recovery model because they participate in an always-on availability group.
• The instance name for an Azure Arc-enabled SQL Managed Instance must be 15 characters or fewer, including any suffix for instance indexing.

Roles and Responsibilities
The responsibilities differ significantly between Azure PaaS services and Azure Arc hybrid services:

• Azure PaaS:
  • Microsoft handles the infrastructure, operations, and provides an SLA.
• Azure Arc Hybrid Services:
  • While Microsoft supplies the software, customers are responsible for the infrastructure and day-to-day operations. Consequently, Microsoft does not offer an SLA for these services since the underlying infrastructure is customer-managed.

Disaster Recovery

This is one of the most dreaded events to face, as it means the Azure Local cluster cannot be recovered and you must restore the workloads either on another cluster or directly in Azure using Azure Site Recovery. On the day you need to perform disaster recovery, your least concern should be that the VMs are not Arc VMs, I’ll spare you repeating that no current system can carry out disaster recovery to Azure or another Azure Local cluster while retaining the VMs as Arc VMs.

Regarding the first option (restoring the workloads in a new Azure Local Cluster), I won’t dwell on it since it simply involves restoring the backed-up components described in previous sections.

The second option involves performing backups and restorations using Azure Site Recovery (currently in Preview). This approach allows you to restore VMs in Azure after an incident or a planned event, and once the incident is resolved, to perform a failback, returning the workload to the Azure Local cluster.

For this, let’s delve into the solution in the following section.

Azure Site Recovery (preview)

Below, I will detail, summarize, and restructure information from the official documentation on Azure Site Recovery (preview), highlighting key factors to consider when using this service.

We start with the most critical aspect: when you want to use Azure Site Recovery on your cluster, it will install an agent on each node. Because this agent is not “properly signed,” WDAC will block its installation, throwing the error “Microsoft Azure Site Recovery Provider installation has failed with exit code - 1.” To overcome this, you must set WDAC to “Audit” mode. For example, you can run the following command on any node in the cluster:

Enable-AsWdacPolicy -Mode Audit

Next, we analyze the disaster recovery strategy. The Azure Site Recovery process for Azure Local involves four key stages:

• Replication: The VM’s virtual hard disk (VHD) is copied to an Azure Storage account.
• Failover: In the event of a disaster, the replicated VM is started in Azure. A test failover can also be run without impacting production.
• Re-protect: After failover, VMs can be re-replicated back from Azure to your on-premises system.
• Failback: Finally, you can fail back to your original on-premises environment.

Important: The current integration does not support replication, failover, or failback for Arc resource bridges or Arc VMs (although the latter point is nuanced – they might not officially support it, but it works).

The documentation details the installation process and available options, so I won’t go into extensive detail for each step. Instead, here’s a simple guide for configuration:

1. Change WDAC to Audit Mode:
   Set WDAC to Audit mode as shown above.

2. Create a Storage Account and VNet:
   Create a Storage Account (using LRS and without SoftDelete for cache purposes) and a virtual network for testing ASR failover, where the workloads will be restored.

3. Prepare the Infrastructure:
   Create a Vault, a Hyper-V site, and a replication policy (using the default settings in our case). This step installs the ASR extension on the cluster nodes.
   

4. Configure Replication:
   Once the extension is installed and the site is configured, proceed to set up replication for your workloads.
   

5. Synchronize the Workloads:
   The workload will begin to synchronize. In my test lab (set up on a laptop), this took nearly one hour.
   

6. Perform Failover Operations:
   Once synchronization is complete, you can execute both test failovers and regular failovers, moving the workloads from Azure Local to Azure directly.
   

As you can see, this solution is a viable option if you use Azure Local solely as an edge infrastructure with a few critical workloads that you want to secure and keep operational in case of an incident. It is also suitable for single-node infrastructures, as it allows continued operation if connectivity and resource access requirements are met post-failover. A notable strength is the re-synchronization of workloads after failover and the possibility to fail back to the cluster once it is restored. Personally, I would love to see an automatic failover mechanism triggered by incidents, though I understand that not every disaster recovery scenario needs to operate automatically. I also miss a mechanism that would allow syncing different Azure Local clusters using Azure Site Recovery, but maybe this will be possible in the future.

Since this solution is in Preview, it isn’t supported in production by Microsoft and may present certain challenges (such as the need to set WDAC to Audit mode). Nevertheless, in my view, it can be an alternative for environments where no backup is available or where high workload availability is critical. Keep in mind that for the workloads to function correctly afterward, additional considerations such as connectivity and access (via load balancers, DNS, VPNs, etc.) must be addressed.

With this, I conclude the section on Disaster Recovery using ASR. There are additional points in the documentation worth reviewing, such as more granular synchronization groups and prioritizing certain workloads.

Conclusion

Much to my regret, this has been an extremely lengthy and somewhat impractical article (at least in terms of lab integration), but I hope you have gained some insights (or at least compared notes) about my experience in this field. If you’d like to add anything or have me correct any points, feel free to contact me directly via LinkedIn.

I hope your experience goes beyond merely configuring a backup, you must test the restores as well. If you don’t test restorations, you can’t be sure your backups work. Given that Azure Local via Arc Bridge is a somewhat complex solution in this regard, I hope you can plan and define the limitations of your backup strategy.

I wish you a very good weekend, and I’m off to work on the next article: Planning, Sizing, and Migration, which will be the final installment of the Chronicloud saga on Azure Local 😊

Welcome to a new chapter in the Chronicloud series, specifically one of the last articles about Azure Local, where we’ll discuss Redundancy. Initially, I wanted to include both backup and disaster recovery in this post, but as I started writing, I realized it was getting huge and unwieldy. Therefore, I decided to separate those topics into their own dedicated article.

The subject of redundancy tends to be tricky when talking with clients, not because they lack interest, but because the complexities and limitations can be daunting. This article might be more technical than usual, so if you’re unfamiliar with any terminology, please reach out to me directly (LinkedIn is great) and I’ll be happy to clarify the concepts.

Because I’m based in Germany, most customers here love having multiple physical data center locations separated by firewall walls, fully independent “rooms” that can operate if one site goes down. This naturally leads to a desire for multi-location redundancy. In the past (with Azure Stack HCI 22H2), that wasn’t a major problem if you only had two sites and no AKS, because a Stretch Cluster could usually do the trick. But with Azure Stack HCI 23H2 (now Azure Local), the integrated Resource Bridge doesn’t support Stretch Clusters, and that has changed the redundancy equation significantly.

In this post, we’ll tackle redundancy from several angles: nodes (physical hardware), storage, network and Active Directory. Let’s get started!

Redundancy

I’m splitting the redundancy discussion into three main parts:

1. Nodes: Physical node redundancy in the cluster. For instance, if you have three nodes and two fail, you’ll see a potential loss of data and cluster availability.
2. Storage: We’ll look at data resiliency on disks, such as how a two-node cluster might keep operating if one node fails, but if the remaining node’s single disk dies, data might still be lost.
3. Other Infrastructure: Network design (no point having multiple nodes if everything relies on a single switch), multi-location data centers (e.g., rack-aware clusters) and Active Directory design (including why nesting domain controllers in the cluster can be problematic).

Throughout, I’ll do some rough capacity and resource calculations using 2 x 1 TB NVMe capacity drives plus 1 TB RAM per node, just as an example.

Redundancy of the Nodes

Talking about node redundancy means asking how many nodes a cluster can have (up to 16) and what happens when multiple nodes fail. In theory, if you have “half plus one” of the nodes functioning, and (or) you have a quorum (Azure File Share in the case of Azure Local), the cluster remains “operational”. However, that doesn’t guarantee that your storage remains accessible if the data resiliency requirements aren’t met, which is especially true with three-way mirrors in clusters of three or more nodes. If enough nodes go down simultaneously, data could become inaccessible or lost.

Personally, I’m not a huge fan of clusters with more than five nodes for Azure Local. The reason is that Storage Spaces Direct (S2D) can handle only so many simultaneous node failures while still keeping the data intact, and I prefer to keep things simpler. Let’s walk through some typical node counts:

One Node

You can technically run Azure Local with just one node. But it isn’t really a “cluster” since you have zero redundancy. If that single node fails, so does your entire environment. This setup might be simple in terms of CPU/RAM/storage planning, but I wouldn’t recommend it for any production environment that needs high availability. Even a minor node failure or an update problem can lead to downtime that’s tricky to fix.

Two Nodes

A two-node cluster gives you basic redundancy. If one node dies, the other automatically takes over the workloads, provided you haven’t over-allocated RAM and CPU. For example, if each node has 1 TB of RAM, you might be tempted to run workloads totaling 2 TB of RAM (like 2 VMs with 700 GB RAM allocated). But in a node-failure scenario, you only have 1 TB left to run everything. You’d see workloads failing to start if they collectively need more than 1 TB.

Overcommitting CPU also has consequences. If you had a 1:3 ratio (physical CPU : vCPU) across both nodes, losing one node means your ratio effectively doubles for the surviving node, possibly halving performance. But at least the cluster remains functional.

Three or More Nodes

With three nodes, you can leverage three-way mirror resiliency for your data, which allows the cluster to keep running if one node fails, even two nodes can fail in clusters that have more than 3 nodes and data still be safe. But if you run out of “mirrored copies” across multiple failures, data can be inaccessible or lost.

Another consideration is resource usage. Suppose each node has 1 TB RAM and you have workloads requiring 2 TB in total, distributed across the three nodes. If one node fails, you have only 2 TB left across two nodes. But if each of those two remaining nodes is already using a large portion of RAM, certain VMs might not be able to start. This is why you need an overprovisioning plan (or “failover capacity plan”) to handle node failures gracefully.

With bigger clusters, more resources are available for workloads, and you get more “hardware-level” redundancy. But S2D is generally tested up to losing two nodes simultaneously. If a third node also fails, you could run into data integrity issues, even though the cluster might still appear to have a majority for quorum. That’s one reason why I tend to keep my Azure Local clusters within a modest node count (like 2 to 5).

Redundancy and Resiliency in the Storage

For raw vs. usable storage capacity calculations, I use the S2D calculator by Cosmos Darwin (LinkedIn): https://aka.ms/s2dcalc

Important: Azure Local requires a minimum of two capacity drives per node (hardware requirements). For two nodes, you’ll generally have a two-way mirror, and for three or more nodes, a three-way mirror. All nodes should have the same number of disks (same capacity and model).

Two-Node Cluster

A two-node cluster with a two-way mirror ensures that if one node fails, the other still hosts the data. When the failed node is restored, the system automatically re-syncs the data. This re-sync can take minutes or hours depending on network bandwidth, RDMA configuration (RoCE/iWARP), and the amount of data that needs to be copied back.

If each node has two 1 TB NVMe disks, that’s 4 TB raw total. However, because it’s mirrored, you only get about 1 TB of usable capacity:

Four-Node Cluster

With four nodes, you use a three-way mirror. This means data is replicated across at least three different nodes, so you can survive up to two node failures at once without losing data. But if a third node fails, you risk losing data consistency.

Re-sync times also get longer with more copies to rebuild. In extreme scenarios involving enormous data volumes, re-syncs could take days, though in my experience it’s usually hours at most. It depends heavily on your network speed, RDMA adapters, and switch capacity.

Microsoft’s fault tolerance documentation for S2D (like this link) provides some scenarios:

• Situations where everything stays online (losing more than two drives but at most two servers):

• Situations where everything goes offline (drives lost in three or more servers at once, or losing three nodes simultaneously):

For example, four nodes each with 4 x 1 TB capacity disks gives you 16 TB raw, but effectively only 4 TB usable under a three-way mirror:

Volume Counts: It’s best practice (see Microsoft’s volume planning guidelines) to match the number of volumes to a multiple of the number of servers. For four servers, create four volumes if you want balanced performance. Each node can “own” one volume’s metadata orchestration.

Other Redundancies in the Infrastructure

We can’t ignore the rest of the infrastructure that supports Azure Local, such as network design and Active Directory architecture. Let’s briefly discuss these points:

Network Redundancies

You can handle Azure Local networking in multiple ways. Some designs feature a dedicated storage switch (especially if you have more than four nodes). You might also want to split nodes across physical data center “rooms,” but official multi-site or campus-cluster setups are currently not supported (“Rack Aware Cluster” is in private preview).

Switches

Well, it depends on the number of nodes, your storage traffic design, and your budget. But you always need at least two switches for management and compute traffic. A single switch would act as a single point of failure, nullifying your otherwise redundant cluster design.

For up to four nodes, you can technically go switchless (often called Direct Attach) for the storage network, meaning the nodes are cabled directly to each other. I recommend RDMA (especially RoCEv2). This setup ensures that if one node fails, the remaining ones maintain inter-node connectivity. Personally, though, I consider switchless challenging beyond two nodes, it requires precise configuration, can be tedious to implement (via ARM templates), and is not very flexible (you can’t easily add new nodes or modify them without major rework). Its one advantage is cost savings if your existing switches can’t handle RDMA traffic. For reference, here’s a link describing that setup in detail: Four-node switchless, two switches, two links

And here’s the official diagram for that cabling method (props to whoever created it!):

In the image, each node has two connections to its neighbors, using six total network ports exclusively for storage. Because most network cards have two ports, you wouldn’t want to use both ports on a single NIC card to connect to the same node. If that NIC card fails, you’d lose the connection to the node entirely. Additionally, note that you can’t easily expand beyond two nodes in a “switchless” scenario, the only supported move is from a single node to a two-node cluster. Anything above two nodes requires a switch, as explained here.

Multi-Location or Rack Aware

After deciding how many switches to use, you also need to determine how many physical “locations” your cluster needs, e.g., is it all in one data center room, or multiple? Those familiar with the product already know this, but newcomers might be surprised to learn that Azure Local (starting 23H2) doesn’t support stretch clusters, meaning nodes should theoretically be in the same rack. I’ll mention a new way to set up Azure Local for two locations (really just two rooms in the same building) called Rack Aware Cluster.

In previous versions, you could configure a so-called stretch cluster for two sites, provided your round-trip latency didn’t exceed 5 ms (see Microsoft docs). However, that’s now invalid because Azure Local 23H2 includes new cluster management elements (like Azure ARC Resource Bridge) that don’t support stretch clustering. In my opinion, there’s also no real need to expand clusters this way because Azure Local is more of an Edge Datacenter solution, not a typical “full” data center approach. If you want balanced infrastructure across two sites, Windows Server 2025 might be a better option, or you could split Azure Local into two separate clusters.

Nevertheless, there is a future possibility (currently in private preview) to meet this need, known as Rack Aware Cluster (TechCommunity article). This upcoming feature may include up to eight nodes (four per side). Its advantage is that it could support the ARC Resource Bridge because the nodes reside in the same Layer 2 network. If I ever get the chance, I’ll test it once a public preview is available. For now, we only have a network diagram:

Active Directory Redundancy

In my opinion, the most controversial part of designing a new Azure Local environment is Active Directory, mainly because there’s no official best-practice guidance covering it. Starting with Azure Local 23H3, you can integrate Azure Local into your existing Active Directory with minimal effort by following the prep steps in this Microsoft doc. Those steps outline the prerequisites for registering Azure Local and creating the cluster. The question is: Do I use my current corporate AD (with all existing data), or do I spin up a separate AD domain (often called the Fabric Domain) just for Azure Local?

Unsurprisingly, the answer is: it depends, it depends on the management model you want and how you plan to operate the cluster. However, in the majority of cases, I recommend creating a dedicated Active Directory domain for Azure Local, so you avoid unexpected permission issues or admins deciding it’s a great idea to apply random GPOs to the solution (I’ve seen some downright bizarre situations arise that were very tough to fix afterwards! 🤣)

Keep in mind that if you create a new Active Directory, it isn’t strictly required to integrate workloads like AVD or AKS. The users relying on those services and the workloads themselves, base their authentication on Entra ID (and, in terms of network, you can just set VLANs to point to whatever domain controllers they specifically need). You should also remember that Azure Local can be managed through Azure RBAC in the portal, making dedicated Fabric Domain accounts largely unnecessary. For now, as long as Azure Local still requires some form of AD domain (the version without AD is apparently coming later this year), my personal recommendation is to stand up a dedicated AD domain for Azure Local.

So, once you decide you want a new AD domain, where do you host it? One possibility is to run a domain controller for that new Fabric Domain in a VM outside Azure Local (for example, in your existing infrastructure or even on a laptop if you’re brave 😜). Then you register Azure Local with Azure and add a second domain controller inside the cluster itself. A frequent question is whether you can nest both DCs in the cluster. Theoretically and practically (if we’re being honest), yes (but you shouldn´t), you can migrate the domain controller you used to register Azure Local into the cluster. The problem arises if there’s a cluster wide outage. The cluster tries to start up, but both domain controllers are also part of that same cluster (even the storage!). You hit the classic chicken-and-egg scenario: the cluster can’t come online because there’s no AD available, and AD can’t come online because the cluster is offline.

Because of this, my one ironclad piece of advice is that no matter what you do (new domain or existing domain), ensure that at least one domain controller remains outside the cluster and is configured as DNS for the cluster. You can nest the second domain controller in the cluster without issue, or even migrate it in as an ARC VM. (We’ll talk more about that migration flow in future articles.)

Conclusion

Although I shortened some sections, I think this article still provides a clear (and somewhat “simple”) overview of what to consider when designing Azure Local with redundancy in mind. I had to omit backup and disaster recovery to avoid tripling the size, but those will be covered in the next post.

The best advice I can give is: if you’re new to Azure Local, reach out to a trusted consultant or your OEM for guidance before diving in. If you’re set on going solo, join the Azure Local Slack channel and clarify any doubts first. It’s easy to paint yourself into a corner if you’re not aware of certain limitations.

Thank you for reading all the way through! If you have any questions, feel free to ping me on LinkedIn. I’m always happy to help clarify or share insights from my experience.

I decided to write this article outside of my usual Chronicloud series, though it does follow on from the Azure Stack HCI: Azure Virtual Desktop post. The reason? Image management for AVD and deploying those images in Azure Local can be a real pain point. Currently, the entire process is quite manual, involves tons of steps, and doesn’t offer much flexibility. So, I thought sharing how to use Nerdio for this might be helpful.

I was also inspired by attending the Nerdio Micro Hack in Frankfurt last Wednesday (kudos to Bas van Kaam (@LinkedIn) and Bjørn M. Riiber (@LinkedIn) for the excellent masterclass). Most attendees seemed to need a solution for running AVD on-prem. While it is indeed possible to manage these images without any extra tooling, just a handful of PowerShell scripts and some Azure DevOps pipelines, I find that approach clunky. Nerdio, on the other hand, does a great job by also offering more flexible autoscaling plans and simple scripted tasks that are far easier to configure.

For those unfamiliar with Nerdio ( getnerdio.com ), it’s a company that specializes in a simple, straightforward way to manage AVD, Windows 365 Cloud PCs, and Intune (no DevOps overhead or IaC expertise needed). Because I want to specifically focus on the Azure Local integration (and there isn’t much up-to-date documentation on this topic), I’m documenting the process here. I won’t be covering the Nerdio installation in your Azure subscription, my friend Michael Frank (@LinkedIn) and Neil McLoughlin (@LinkedIn) have you covered:

• Michael’s blog on how to install Nerdio: Blog Link
• Neil’s YouTube video on setup and basic config:

I also won’t dive deep into FSLogix here. I’ll save that for another article in the Chronicloud series when we talk more about AVD.

The only official documentation I’ve found so far for using Azure Local with Nerdio is:

• Azure Stack HCI and Nerdio Manager
• Troubleshoot Issues with Resource Bridge in Azure Stack HCI

Below is a diagram illustrating the process for registering Nerdio and deploying AVD on Azure Local through Nerdio:

And here’s a reference architecture of the infrastructure which I´m using:

Configuring Nerdio for Azure Local

If you take a look at the earlier documentation, you’ll notice it mentions creating a custom location. This is actually no longer necessary because Azure Local now automatically generates a custom location during its deployment process. I believe the existing documentation hasn’t been fully updated to reflect this (and I don’t blame anyone, Azure Local evolves so quickly that staying on top of changes is a challenge, and I face the same issue with my own blog posts).

Before we dive into the configuration, there are a few prerequisites to meet. Thankfully, most of them were already taken care of in the article “Azure Stack HCI: Azure Virtual Desktop”:

• Install Nerdio Manager: This should be done using the guides mentioned in the introduction.
• An Active Directory for the users who will be using AVD (with identities synced to Microsoft Entra ID).
• A fully functional Azure Local cluster (at least 1 node), registered with an Azure subscription that is connected to Nerdio Manager.
• Azure Arc registration for the Azure Local nodes.
• Resource Bridge already enabled in Azure Local.
• A local shared storage path on the Azure Local cluster for FSLogix profiles.
• The logical network and resource group where your Azure Local cluster is registered must be linked.

Registering Resources in Nerdio

Once Nerdio is installed, you’ll need Owner permissions on the resources you’re about to register, using the same account that performs the registration in Nerdio. The resources you’ll register are:

• Subscription
• Resource Group
• Hybrid Network (a.k.a. your Logical Network)

You can register all of them from the same management window under Settings > Azure environment:

You’ll also need to connect your Active Directory (with a user who has the rights to join machines in the appropriate OU) and specify the file share for FSLogix profiles (in my case, a file share hosted on the domain controller(don´t replicate 😅)). This is done in Settings > Integration:

Golden Image

The process of creating a Golden Image was outlined in the introduction, and if you’ve worked with Nerdio before, you’ll find it’s a fairly straightforward process. Here’s my quick rundown:

1. Set up a virtual network (vNet) in Azure (not your local, logical network). I created a simple vNet in the same Resource Group as my Azure Local cluster, changing only the name. You can do this in the Azure portal.

2. Import the VM image from the Azure gallery by going to Desktop Images > Add from Azure library. Select both the vNet you just created and the Azure image you want to use as your Golden Image. My setup looked like this:

Nerdio will then spend about 30 minutes creating a VM based on the parameters you provided. It places that VM in the specified vNet and Resource Group. By default, this VM won’t be accessible from the internet unless you assign it a public IP and configure RDP access in the Azure portal.

Now, you can modify this VM in two ways:

• Direct Access: Connect to the VM (via VPN or public IP) to configure it manually.
• Run Scripts: One of my favorite features in Nerdio. You can run scripts to install language packs or other software, either on demand or scheduled. (If you’re thinking what I’m thinking, this is perfect for scheduling and automating Windows Updates on your Golden Images!)

That sets up your base VM to serve as your Golden Image. Next comes the more interesting part: deploying the image into a Compute Gallery and syncing (or uploading) it to Azure Local so it can be used on your Session Hosts.

You can accomplish this by converting the VM into an Image and syncing it with Azure Local like so:

As you can see in the last screenshot, I’ve enabled Geographic distribution & Azure compute gallery because it allows me to pull the image down to Azure Local by selecting my custom location. In my case, I created the Compute Gallery automatically from within Nerdio and selected the custom location azlfra. If you wish, you can also configure a schedule so that this process runs automatically the day after Windows Updates are installed on the Golden Image:

It took about 30 minutes for my image to sync. I really appreciate how Nerdio keeps you updated on the entire progress. Here’s what the result looks like:

And that’s it! You now have a Golden Image you can generate periodically and automatically update through Nerdio. It saves you a ton of time and makes image management much more efficient and user-friendly.

Creating a Workspace

We now have an image in Azure Local that we can use to provision a Session Host. So, what’s next? Since I’m starting from a clean deployment, I’ll create a Workspace dedicated to Azure Local, where I’ll implement a “Hybrid host pool”. I use quotes because, realistically, these host pools shouldn’t be mixed between on-prem and cloud (not supported yet), so they’ll reside solely in Azure Local.

Adding a Hybrid Host Pool

After the Workspace is created, head to Hybrid host pools and add a new one. In my case, I chose Multi user desktop (pooled). You’ll need to select the Active Directory and FSLogix profile you configured under Integrations, pick the hybrid network you set up in the Azure Environment settings, and finally select the Desktop Image we created earlier. I assigned three users manually via Quick Assign, but in a production environment, you’d typically assign users through security groups.

Configuring Autoscaling

Once the host pool is created, you’ll see a prompt to configure Autoscale. This differs from “Scaling Plans” in the Azure portal, which mainly handle power management (the dynamic autoscaling feature is still in preview). Nerdio offers a far more robust approach to autoscaling, with several factors to consider:

• HOST POOL PROPERTIES
  Specifies max sessions per Session Host and how load balancing is handled.

• HOST POOL SIZING
  Determines when a Session Host is considered active (based on whether the VM is running or if the AVD agent is available), sets a base capacity for Session Hosts, a minimum capacity, and a “bursting” feature that automatically creates Session Hosts to meet demand.

• SCALING LOGIC
  Lets you use multiple triggers (in parallel if you wish) to scale in and out Session Hosts. For Azure Local, there are currently only three autoscale triggers:
  1. Avg active sessions: Scales out when the average number of active sessions exceeds a predefined value.
  2. Available sessions: Maintains a specified number of available hosts by scaling in/out within the limits of Host Pool Sizing and max sessions per host.
  3. User-driven: Starts hosts when users connect, then automatically deallocates them after a set time of zero user sessions.

• ROLLING DRAIN MODE
  Lets you create multiple “drain windows” and target a percentage of your hosts to place in drain mode.

• PRE-STAGE HOSTS
  Powers on or creates Session Hosts before business hours, avoiding massive logon storms.

• MESSAGING
  Allows you to notify users of maintenance or shutdown periods. Changing the time automatically updates the in-session notification.

• AUTO-HEAL BROKEN HOSTS
  A “killer feature” of Nerdio. If a Session Host VM isn’t recognized as Available by the AVD service, Nerdio Manager can try repairing it automatically. It performs this auto-heal after all scale-out tasks are finished and things are stable.

Upon configuring autoscaling, Nerdio provisions a Session Host according to your settings. In my test, the process took roughly 50 minutes and (to my surprise) completed without any errors. Here’s a snapshot of the provisioning and its final result:

Post-Deployment Tweaks

After the deployment, I realized I forgot to install the multimedia redirection extension and enable RDP Shortpath in my Golden Image. Ordinarily, I’d have to update the Golden Image and roll out a new deployment. But with Nerdio, I can use Run Script to fix these settings directly on the Session Host (or multiple Session Hosts simultaneously via bulk operations). Of course, I’ve also integrated these features into my Golden Image as well, but below are two screenshots showing how I added them directly to the Session Host:

Once everything was in place, I logged into the Virtual Desktop, and sure enough, it performed exactly as expected:

Conclusion

A process that would normally take three to four hours to handle manually can easily be automated with Nerdio in about one or two hours at most. The main advantages come from the ability to automate nearly every step, like managing images, generating them automatically after Windows Updates are applied and running scripts to install applications on the fly.

Being able to implement unconventional autoscaling plans is another standout factor of this solution. Although I do wish we had a few extra scaling triggers on Azure Local (for CPU and RAM metrics), the current setup is already quite impressive compared to the Standard processes.

Bear in mind, Azure Virtual Desktop on Azure Local is a pay-as-you-go model (you pay for the Session Hosts while they’re running), so Nerdio’s autoscaling capabilities can help optimize costs. Granted, the cost savings aren’t as pronounced as on Azure Public, but the flexibility and ease of management still make it well worth considering.

Personally, I’m impressed by how straightforward and effective Nerdio is in conjunction with Azure Local. While I’m no stranger to Nerdio in general, seeing it work with Azure Local reinforces how viable this combination can be, especially for admins who want a comprehensive solution without needing deep technical knowledge in DevOps or infrastructure as code.

Welcome back to another article about Azure Local, our seventh in the series! (It’s starting to shape up quite nicely, isn’t it? 😊). This time, we’ll dive into a topic I’ve discussed with several colleagues and one that has sparked a fair share of questions: Lifecycle Management. A question that often pops up is whether this process belongs under the umbrella of Day 2 Operations 🤔.

In my opinion (and hey, I could be wrong!), Lifecycle Management deserves its own distinct category within Day 2 Operations. When planning earlier articles, I avoided diving too deep into this area to keep things concise. However, I felt it warranted a dedicated article to give it the attention it truly deserves.

My goal with this post is to walk you through these processes in a clear and straightforward way, helping to demystify some of the perceived complexities around Azure Local Lifecycle Management.

Let’s get started!

Updates

The update process in Azure Local (starting with version 23H2) is straightforward and easy to manage from an administrative perspective. In this section, I’ll walk you through how this process works, its phases, and the key considerations to keep in mind before applying updates.

Spoiler alert: The information I’m sharing is a mix of official documentation (from Microsoft and some OEMs) and my personal experience since the product launched in February 2024. For the sake of brevity, I won’t dive into specific scenarios like SDN or updates for Stretch Clusters, and yes, while Stretch Clusters aren’t available in this version, there’s a Rack-Aware option, which I also won’t cover here.

Azure Local Updates

I recently updated my Azure Local cluster, and I want to share my experience based on the article About updates for Azure Local 23H2. Here’s everything you need to know to keep your system up to date.

Keeping the Cluster Updated

Let’s face it, keeping your system updated is critical, especially with the new Azure Local 23H2 update process. This version introduces the Lifecycle Manager (orchestrator), which simplifies how updates are deployed and managed for the OS, agents, services, and even drivers and firmware.

If you’re still running Azure Stack HCI 22H2, be aware that support ends in May 2025. I recommend making the move to 23H2 sooner rather than later to ensure your system remains secure and supported.

Why the New Update System Is Better

Here’s what stood out to me during the update process:

• Simplified management: The orchestrator consolidates all updates, from the OS to hardware-specific drivers, into a single workflow.
• Peace of mind: Automatic health checks run before and during updates, minimizing downtime and avoiding disruptions.
• Resilience: Any issues during updates are retried and resolved automatically.
• Consistency: Whether you manage updates locally or through the Azure portal, the experience is intuitive and efficient.

The system is designed to make updates easier and save you time. It’s been a significant improvement from previous versions.

What’s Included in the Updates?

When I updated my cluster, these were the main components:

1. Azure Stack HCI OS: Security patches and reliability fixes keep the system stable and productive.
2. Agents and Services: The orchestrator handles updates for core services, including the Azure Connected Machine Agent and Arc Resource Bridge.
3. Drivers and Firmware: If supported by your hardware vendor, these updates are integrated and automatically installed during the same maintenance window.

This all-in-one approach means less manual effort and fewer maintenance windows. However, keep in mind that customer workloads are not part of this update process.

Update Cadence You Should Know

Updates follow a clear schedule:

• Monthly updates: Regular patches for performance and security.
• Baseline updates: Released quarterly, these include new features and improvements.
• Hotfixes: Emergency fixes for critical issues.
• Solution Builder Extensions: Vendor-specific updates like drivers and firmware.

To stay supported, make sure your system is updated within six months of the latest baseline. Falling behind could leave your system out of compliance.

How I Applied the Updates

Personally, I used the Azure portal to manage the updates. The Azure Update Manager made the process straightforward, with clear steps and a simple interface.

That said, it’s also possible to apply updates using PowerShell, which works well for both single-node and multi-node systems. If you prefer a command-line approach, PowerShell is an excellent alternative. Just avoid unsupported tools like Windows Admin Center, SConfig, or third-party tools, as they will cause issues and might even impact billing (at least as far as support tickets are concerned 😅).

Update Phases

Updating Azure Local to version 23H2 introduces a phased process designed to ensure smooth updates with minimal disruption. Below, I’ll guide you through the phases of the update process based on Microsoft’s documentation and my own experience.

About Update Phases

The update process in Azure Local focuses on keeping the system available while applying updates to operating systems, agents, services and solution extensions. Updates are automated and designed to handle workloads dynamically to maintain uptime. They fall into two categories:

• Updates not requiring reboots: Applied without restarting the system.
• Updates requiring reboots: Use Cluster-Aware Updating to restart machines one by one, ensuring availability.

Updates proceed in the following phases: Discovery and acquisition, Readiness checks and staging, and Installation progress and monitoring. Each phase involves specific steps that may or may not require manual input.

Phase 1: Discovery and Acquisition

Before an update is released, Microsoft validates it as a package of components. Once validated, release notes are published detailing:

• Update contents
• Changes introduced
• Known issues
• Links to external downloads (e.g., drivers or firmware)

Your Azure Local update platform will automatically detect new updates, but you’ll need to visit the Updates page in your management interface to view details. Depending on your hardware and the scope of the update bundle, additional content (e.g., OEM-specific drivers) might need to be downloaded (SBE Updates). If extra steps are required, the system will prompt you.

Phase 2: Readiness Checks and Staging

Before installing an update, Azure Local performs a series of prechecks to ensure the system is safe to update. These checks cover:

• Storage systems
• Failover cluster requirements
• Remote management
• Solution extensions

These checks identify potential issues that could block or delay updates:

• Blocking conditions: Must be resolved before proceeding with the update.
• Warnings: Could lead to longer update times or workload impact. You may need to acknowledge these warnings before moving forward.

New checks may be added with each update, so it’s critical to run readiness checks after downloading the update package. Note that in this release, updates cannot be scheduled, they must be installed immediately.

Phase 3: Installation Progress and Monitoring

During installation, you can monitor progress via your chosen interface. The update workflow is displayed hierarchically, showing each step as it occurs. Steps may dynamically adjust depending on the update flow.

The update solution includes retry and remediation logic, automatically addressing issues where possible. However, manual intervention may sometimes be required. If you encounter a blocking issue, you’ll need to fix it and rerun the readiness checks before continuing.

Updates are designed to be as seamless as possible, but the system will prompt you if any critical action is needed to keep the process moving forward.

By following these phases, Azure Local ensures that updates are applied smoothly and reliably, minimizing downtime and keeping your infrastructure up to date.

SBE Updates

The Solution Builder Extension (SBE) is a critical part of managing and maintaining Azure Local, especially when it comes to ensuring your hardware is up to date. Based on Microsoft’s documentation and insights shared by Jaromir in his Lifecycle Manager Deep Dive, let’s explore what SBE updates bring to the table and how they improve the lifecycle management process.

What Are SBE Updates?

The SBE, referred to as Solution Builder Extension in Azure CLI, allows you to apply updates from your hardware vendor to your Azure Local system. These updates are an integral part of Azure Local 23H2, where SBE updates are packaged into a unified solution update. They include:

• Drivers and firmware updates: Released by hardware vendors to enhance hardware compatibility and performance.
• Hardware monitoring enhancements: Tools and diagnostics to maintain hardware health.
• Advanced pre-update checks: Custom validation logic integrated into Azure Local’s health checks.

With these features, SBE enables streamlined updates and introduces vendor-specific enhancements to your Azure Local system.

How SBE Updates Work

Starting with Azure Local 23H2, SBE updates are integrated directly into the Lifecycle Manager. They can be applied as part of a combined solution update or as standalone updates. When an SBE update matching your system’s hardware is available, it appears in the Azure portal or can be retrieved using PowerShell.

SBE updates also include advanced capabilities, such as:

• Health service integration: Extending pre-update health checks to evaluate hardware issues (e.g., power supply failures, SSD wear, or software issues).
• Download management: Allowing Azure Local to download and apply future updates automatically, reducing manual effort.
• Customized updates: Vendor-specific steps before and after the solution update process, ensuring hardware compatibility.

Key Considerations for SBE Updates

1. Integration with Lifecycle Manager: Azure Local’s orchestrator ensures that SBE updates are part of the same workflow as OS and service updates.
2. Manual intervention for older systems: For hardware not supporting the SBE update experience, updates may need to be applied separately using tools like Windows Admin Center.
3. Health checks before updates: SBE integrates into Azure Local’s readiness checks, ensuring any hardware-related issues are addressed before updates proceed.
4. Supported hardware: Newer integrated systems and Premier Solution hardware fully support SBE updates. For older hardware, updates might require manual handling.

Additional Insights from Jaromir’s Deep Dive

Jaromir’s Lifecycle Manager Deep Dive provides an excellent breakdown of the SBE update process. While the implementation details are not included here, the guide discusses:

• The structure and packaging of SBE updates.
• How to manage sideloading for environments with restricted connectivity.
• Troubleshooting issues like failed updates or health check prerequisites.

For further guidance, I highly recommend checking out Jaromir’s work if you need more in-depth insights into handling SBE updates effectively.

Upgrade

The Upgrade process in Azure Local encompasses several tasks that go beyond standard updates. While some actions, like upgrading extensions might feel routine, others, such as scaling up nodes or transitioning from Azure Stack HCI 22H2 to Azure Local 23H2, require more deliberate effort and planning.

Many of these processes aren’t strictly classified as updates but play a critical role in maintaining, scaling, and enhancing the functionality of your Azure Local infrastructure. In this section, I’ll introduce each upgrade process and set the stage for a deeper dive into the specifics in subsequent sections.

Here’s what we’ll cover:

• Extensions: Upgrades for both automatically installed extensions during cluster deployment and manually added ones.
• Add a Node: Expanding cluster capacity by adding new nodes to increase CPU and RAM resources.
• Add Storage: Scaling storage capacity by adding disks to nodes and expanding the storage pool.
• Repair a Node & Extensions: Replacing defective nodes and restoring critical extensions for Azure Local management.
• Secret Rotation: Updating administrative passwords and the secret for Azure Arc Resource Bridge to maintain security.
• Upgrade from Azure Stack HCI 22H2: A detailed and challenging process involving the transition to Azure Local 23H2 with careful dependency and compatibility analysis.

Each of these upgrade steps serves a unique purpose in optimizing and evolving your Azure Local environment. In the following sections, we’ll explore them one by one, providing detailed instructions and insights for each process.

Extensions

Managing extensions in Azure Local is a crucial part of keeping your cluster up to date and ensuring it runs optimally. Azure Local supports both Azure-managed extensions and customer-managed extensions, and upgrading them is a seamless process thanks to Azure Arc integration. Below, I’ll break down how the update process works based on Azure’s documentation.

How Extensions Are Upgraded

Extensions in Azure Local can be updated automatically or manually, depending on the type of extension and its configuration. By default, automatic upgrades are enabled for most extensions, provided the extension publisher supports it. Let’s explore the two main types of extensions and their upgrade methods:

1. Azure-Managed Extensions:
   • These are installed automatically when your Azure Local cluster is registered with Azure. They are essential for the platform’s functionality and include extensions like telemetry, diagnostics, and the Remote Support Arc extension.
   • Upgrade Process:
     • Azure-managed extensions are typically updated as part of the solution update process.
     • If these extensions were missing when the cluster was registered, Azure will display a banner on the Extensions page guiding you to install them.
2. Customer-Managed Extensions:
   • These extensions, such as Azure Monitoring Agent, Azure Site Recovery, and Windows Admin Center, can be manually installed and managed.
   • Upgrade Process:
     • Automatic upgrades are enabled by default if supported by the extension publisher.
     • Manual upgrades are performed via the Azure portal, allowing you to select the desired version and apply it across the cluster.

Automatic Extension Upgrades

Automatic extension upgrades simplify maintenance by ensuring extensions are updated without manual intervention. Here’s how the process works:

• When a new version of an extension is published, Azure deploys the update in batches across regions and subscriptions.
• If the extension fails to upgrade, Azure retries the process or rolls back to the previous version automatically.
• Extensions with multiple upgrades can be processed in batches, but each upgrade is applied individually to ensure consistency.

Manual Extension Upgrades

In scenarios where automatic upgrades are disabled or a specific version needs to be installed, you can perform a manual upgrade:

1. Navigate to the Extensions page in the Azure portal.
2. Select the extension you want to upgrade and choose Settings.
3. Choose the desired version and click Save.

This approach is useful for:

• Addressing version mismatches across nodes.
• Testing new extension versions before enabling them across the cluster.

Key Considerations

• Cluster-Wide Operations: Upgrading an extension is a cluster-aware operation. Once updated, the extension is automatically applied to all nodes, including new nodes added to the system.
• Rollback and Retry: Azure’s platform ensures failed upgrades are retried or rolled back to avoid disruptions.
• Manual Intervention: If extensions repeatedly fail to upgrade, you can disable automatic upgrades to troubleshoot the issue before re-enabling them.

Extensions are an integral part of Azure Local’s functionality, and their upgrade process, whether automatic or manual, ensures your cluster remains aligned with the latest features and updates.

Add a Node

Adding a node to your Azure Local cluster is one of the most straightforward ways to scale up resources like compute and storage. While I don’t currently have the infrastructure to reproduce this process step by step, I’ve successfully carried it out in the past, transitioning from a single-node to a two-node system without any issues. Based on my experience and this article from Microsoft, let me guide you through the process.

Overview

Azure Local supports scaling from 1 to 16 nodes, allowing you to increase compute and storage dynamically. Adding a node involves preparing the new hardware, validating its compatibility, and integrating it into the existing system. The orchestrator, also known as the Lifecycle Manager, ensures that everything from network configurations to storage resiliency adjusts seamlessly during the process.

Here are some important highlights:

• Node Requirements: The new node must closely match the existing cluster in terms of CPU type, memory, number of drives, and drive specifications.
• Scaling Limitations: In this release, nodes must be added one at a time. Multiple nodes can be added sequentially, but storage rebalancing is triggered only once at the end.

Key Steps to Add a Node

1. Prepare the Node:
   • Install the operating system and required drivers on the new node. Follow the guidance for Installing the Azure Local Operating System, version 23H2.
   • Register the new node with Azure Arc. Ensure the same parameters (e.g., Resource Group, Region, Subscription, Tenant) as the existing nodes are used.
   • Assign the following permissions:
     • Azure Local Device Management Role
     • Key Vault Secrets User
2. Add the Node Using PowerShell:

   • Sign in to the existing node with AzureStackLCMUser credentials (or another user with equivalent permissions).

   • (Optional) Update the authentication token for the orchestrator:

     Update-AuthenticationToken
     

   • For systems running a version prior to 2405.3, clean up conflicting files on the new node:

     Get-ChildItem -Path "$env:SystemDrive\NugetStore" -Exclude Microsoft.AzureStack.Solution.LCMControllerWinService*,Microsoft.AzureStack.Role.Deployment.Service* | Remove-Item -Recurse -Force
     

   • Run the following command to add the node to the cluster. Replace <IPv4> with the new node’s IP address and <Name> with the node’s hostname:

     $HostIpv4 = "<IPv4 for the new node>"
     $Cred = Get-Credential
     Add-Server -Name "<Name of the new node>" -HostIpv4 $HostIpv4 -LocalAdminCredential $Cred
     

   • Make a note of the operation ID provided as output. This ID will be used to monitor the progress of the operation.
3. Validate the System:
   • Use the operation ID to track the status of the process:

     $ID = "<Operation ID>"
     Start-MonitoringActionplanInstanceToComplete -actionPlanInstanceID $ID
     

   • Once the node has been added, the orchestrator begins rebalancing the storage pool. You can monitor the progress of this process using:

     Get-VirtualDisk | Get-StorageJob
     

   • If the rebalancing is complete, the above command will return no output.
4. Sync the Node with Azure:
   • The new node will appear in the Azure portal within a few hours. To expedite this, you can manually sync:

     Sync-AzureStackHCI
     

Supported Scenarios

Azure Local supports adding nodes across various scenarios:

• Single-node to two-node: Transitioning from a single-node to a two-node system requires configuring a witness for resiliency.
• Two-node to three-node: In this scenario, storage resiliency shifts from a two-way mirror to a three-way mirror.
• Three-node to N-node: Supports scaling up to 16 nodes while maintaining a three-way mirror resiliency.

Important Considerations

• Manual Adjustments: If custom storage IPs are used, these must be assigned manually to the new node’s network adapters after it’s added.
• Recovery Scenarios: If the operation partially succeeds or fails, you can use the orchestrator to rerun the operation or repair the node if needed.
• Hardware Validation: The system will block the operation if the new node doesn’t meet storage requirements, but CPU and memory discrepancies will only generate warnings.

Add Storage

Expanding your storage capacity in Azure Local is an essential step to accommodate growing workloads. This process, often called scaling up, involves adding drives to your nodes to increase storage capacity and, in some cases, improve performance. Azure Local relies on Storage Spaces Direct (S2D) for storage management, which simplifies the process by automatically detecting and integrating new drives. Based on Microsoft’s documentation, here’s how to add storage drives to your Azure Local cluster.

Overview of Adding Drives

Adding drives enables you to scale your Azure Local instance without adding new servers. Here’s what to keep in mind:

• Uniformity: It’s highly recommended that all nodes have identical storage configurations to ensure consistency and performance.
• Dynamic Integration: Once new drives are connected and discovered by Windows, they are automatically pooled by S2D and the data is rebalanced across the storage pool.

Key Steps to Add Storage

1. Physically Install Drives:
   • Insert the new drives into the available slots in each server.
   • Ensure that the drives are securely connected and meet the system’s hardware requirements.
2. Verify Drive Detection:
   • Use the following PowerShell command to ensure that the new drives are detected and available for pooling:

     Get-PhysicalDisk | Select SerialNumber, CanPool, CannotPoolReason
     

   • Look for drives where CanPool = True. If CanPool = False, check the CannotPoolReason property to identify the issue.
3. Automatic Pooling:
   • If your system has only one storage pool, S2D will automatically claim eligible drives, add them to the storage pool, and begin redistributing volumes across the drives.
   • If the drives are not automatically claimed, ensure that the system has detected them by manually scanning for hardware changes in Device Manager or using the following PowerShell command:

     Get-Disk | Where-Object IsOffline -eq $true | Set-Disk -IsOffline $false
     

4. Reset Drives with Old Metadata:
   • If the new drives contain old data or metadata, reset them using the Reset-PhysicalDisk cmdlet:

     Reset-PhysicalDisk -FriendlyName "<DriveFriendlyName>"
     

5. Manual Pooling for Multiple Pools:
   • If you’ve configured multiple pools, manually add the drives to the desired pool using the Add-PhysicalDisk cmdlet:

     Get-PhysicalDisk | Where-Object CanPool -eq $true | Add-PhysicalDisk -StoragePoolFriendlyName "<PoolName>"
     

Drive Optimization

After adding drives, data redistribution may become uneven across the pool. S2D automatically optimizes drive usage 15 minutes after new drives are added, using background operations to rebalance the data. This ensures even distribution and prevents certain drives from filling up while others remain underutilized.

• Monitor Optimization Progress: Use the following command to monitor the progress of optimization jobs:

  Get-StorageJob
  

• Manually Trigger Optimization: If needed, you can manually optimize the storage pool using:

  Get-StoragePool "<PoolName>" | Optimize-StoragePool
  

  Important Considerations

• Storage Pool Resiliency: While new drives are integrated seamlessly, ensure the resiliency settings of your existing volumes align with your storage requirements. Adjustments to resiliency settings are outside the scope of this guide.
• Background Operations: Rebalancing is a low-priority task that can take hours or even days to complete, depending on the size and number of drives in your cluster.

Repair a Node & Extensions

In any Azure Local deployment, hardware failures or inconsistent extension states are inevitable over time. To avoid such scenarios, I will provide you with a detailed guide based on Microsoft’s documentation and my personal experience.

Repair a Node

Repairing a node involves bringing a faulty or damaged node back into the system with its original configuration and functionality. This can include hardware replacement, reimaging, or re-registering the node with Azure Arc.

Key Steps to Repair a Node

1. Prepare the Node for Repair:
   • If possible, shut down the faulty node gracefully. Depending on the node’s state, this may not always be feasible.
   • Reimage the node with the Azure Local Operating System (version 23H2) and ensure all required drivers are installed.
   • If custom storage IPs were used during deployment, manually assign the appropriate IPs to the storage network adapters after reimaging.
2. Register the Node with Azure Arc:
   • Use the same parameters (e.g., Resource Group name, Region, Subscription, Tenant) as the existing nodes.
   • Assign the following permissions to the repaired node:
     • Azure Local Device Management Role
     • Key Vault Secrets User
3. Run the Repair-Server Command:
   • On an existing cluster node, sign in with domain credentials (e.g., AzureStackLCMUser) and execute the following PowerShell commands:
     • (For versions prior to 2405.3) Clean up conflicting files:

       Get-ChildItem -Path "$env:SystemDrive\NugetStore" -Exclude Microsoft.AzureStack.Solution.LCMControllerWinService*,Microsoft.AzureStack.Role.Deployment.Service* | Remove-Item -Recurse -Force
       

     • Run the repair operation:

       $Cred = Get-Credential
       Repair-Server -Name "<Name of the new node>" -LocalAdminCredential $Cred
       

       Replace <Name of the new node> with the node’s NetBIOS name.

   • Make a note of the Operation ID provided in the output for monitoring purposes.
4. Monitor the Repair Progress:
   • Use the following command to monitor the operation’s progress:

     $ID = "<Operation ID>"
     Start-MonitoringActionplanInstanceToComplete -actionPlanInstanceID $ID
     

   • Once the repair is complete, storage rebalancing will occur automatically in the background. Monitor this using:

     Get-VirtualDisk | Get-StorageJob
     

   • If no output is returned, the rebalancing process is complete.
5. Recovery Scenarios:
   • If the repair operation fails, you can retry it using:

     Repair-Server -Rerun
     

   • If the operation succeeds partially but a fresh operating system install is required, the orchestrator will handle the reconfiguration.

Repair Extensions

In addition to repairing nodes, maintaining the health of extensions is equally critical.

To simplify the detection and repair of extensions, I created a PowerShell script that automates the process. This script:

• Detects whether the necessary extensions for Azure Local are installed and functional.
• Repairs or reinstalls extensions that are in an inconsistent state.

You can find the script on my GitHub repository: Troubleshooting Extensions Script.

Secret Rotation

Although secret rotation isn’t strictly part of an upgrade process, it is deeply tied to Lifecycle Management. Since we’ve already covered topics like node and extension repairs in this section, I’ve taken the liberty of including secret rotation as well. Ensuring that administrative credentials and service principal secrets are periodically updated is a critical security measure for any Azure Local deployment. Based on Microsoft’s documentation, here’s how to manage secret rotation.

Rotating the Deployment User Password

The deployment user, commonly referred to as AzureStackLCMUser, is responsible for managing your Azure Local environment. Rotating this password can be done with the Set-AzureStackLCMUserPassword cmdlet.

Steps to Change the Deployment User Password:

1. Open a PowerShell session and set the old and new passwords:

   $old_pass = ConvertTo-SecureString "<Old password>" -AsPlainText -Force
   $new_pass = ConvertTo-SecureString "<New password>" -AsPlainText -Force
   

2. Run the Set-AzureStackLCMUserPassword cmdlet to update the password:

   Set-AzureStackLCMUserPassword -Identity mgmt -OldPassword $old_pass -NewPassword $new_pass -UpdateAD
   

3. After executing the command, you’ll see the following warning:

   “The current session will be unresponsive once this command completes. You will have to log in again with updated credentials.”

   Close the session and log in again with the updated password.

Updating the Deployment Service Principal (if you upgrade from 2306)

If you upgraded from Azure Local 2306 to 23H2, you may need to rotate the deployment service principal used during the initial deployment.

Steps to Update the Service Principal:

1. Log into Microsoft Entra ID (Azure AD) and locate the service principal used for the deployment.
2. Create a new client secret for the service principal and make a note of:
   • The App ID for the existing service principal.
   • The new client secret.
3. On one of your Azure Local machines, sign in to Azure using PowerShell:

   Connect-AzAccount
   Set-AzContext -Subscription <Subscription ID>
   

4. Update the service principal name:

   cd "C:\Program Files\WindowsPowerShell\Modules\Microsoft.AS.ArcIntegration"
   Import-Module Microsoft.AS.ArcIntegration.psm1 -Force
   $secretText = ConvertTo-SecureString -String "<client secret>" -AsPlainText -Force
   Update-ServicePrincipalName -AppId <AppID> -SecureSecretText $secretText
   

Changing the Azure Resource Bridge (ARB) Service Principal Secret

The Azure Resource Bridge (ARB) uses a service principal to interact with Azure. Rotating its secret ensures secure connectivity between your on-premises infrastructure and Azure.

Steps to Change the ARB Service Principal Secret:

1. Log into Microsoft Entra ID and locate the ARB service principal. The name typically follows this format: ClusterNameXX.arb.
2. Create a new client secret for the service principal and make a note of:
   • The App ID.
   • The new client secret.
3. On one of your Azure Local machines, run the following PowerShell command:

   $SubscriptionId = "<Subscription ID>"
   $TenantId = "<Tenant ID>"
   $AppId = "<Application ID>"
   $secretText = "<Client secret>"
   $NewPassword = ConvertTo-SecureString -String $secretText -AsPlainText -Force
   Set-AzureStackRPSpCredential -SubscriptionID $SubscriptionId -TenantID $TenantId -AppId $AppId -NewPassword $NewPassword
   

Upgrade from Azure Stack HCI 22H2

The transition from Azure Stack HCI 22H2 to Azure Local 23H2 is a significant upgrade that ensures your system remains secure, supported, and up to date with the latest features. Although I haven’t personally performed this upgrade yet (for better or worse), I’ve followed multiple discussions about this topic in the Azure Local Slack channel. Based on these threads and Microsoft’s documentation, here’s a guide to the process.

It’s important to note that support for 22H2 ends in May 2025, so I highly recommend upgrading your system sooner rather than later.

Supported Workloads and Configurations

Before beginning the upgrade, keep the following points in mind:

• Consult your hardware OEM before you upgrade Azure Local. Validate that your OEM supports the version and the upgrade.
• Upgrading your Azure Local from version 22H2 is only supported for regions where Azure Local, version 23H2 is available. For more information, see Azure Local region availability.
• Use of 3rd party tools to install upgrades is not supported.

Azure Local upgrade supports the following services and workloads:

High-Level Workflow for the OS Upgrade

The upgrade process follows these major steps:

1. Complete prerequisites.
2. Connect to Azure Local 22H2.
3. Install the new OS using PowerShell.
4. Check the status of updates.
5. Perform post-OS upgrade steps.
6. Validate readiness for the solution upgrade.
7. Install the solution upgrade.

Step 1: Complete Prerequisites

Before starting the upgrade:

• Ensure your Azure Local instance is running version 22H2.
• Verify that the system is registered in Azure and all machines are healthy and online.
• Obtain the Azure Local 23H2 OS update. This is available via:
  • Windows Update, or
  • A downloadable ISO from the Azure portal (required if the system lacks internet connectivity).
• Confirm that the client used for the upgrade has PowerShell 5.0 or later installed.

Step 2: Connect to Azure Local

1. Open a PowerShell session on your client machine as Administrator.
2. Establish a remote session to one of the machines in your Azure Local instance:

   $cred = Get-Credential
   Enter-PSSession -ComputerName "<Computer IP>" -Credential $cred
   

Example output:

PS C:\Users\Administrator> $cred = Get-Credential
cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
PS C:\Users\Administrator> Enter-PSSession -ComputerName "100.100.100.10" -Credential $cred
[100.100.100.10]: PS C:\Users\Administrator\Documents>

Step 3: Install the New OS Using PowerShell

To perform the upgrade, follow these steps:

1. Prepare the system:
   • Enable PowerShell remoting:

     Set-WSManQuickConfig
     Enable-PSRemoting
     

   • Verify that the Cluster-Aware Updating (CAU) role is installed:

     Test-CauSetup -ClusterName <SystemName>
     

   • Validate cluster health:

     Test-Cluster
     

2. Check for available updates:

   Invoke-CauScan -ClusterName <SystemName> -CauPluginName "Microsoft.RollingUpgradePlugin" -CauPluginArguments @{'WuConnected'='true';} -Verbose | fl *
   

   Ensure that all machines in the system are offered the same feature update.

3. Install updates:
   • Run the upgrade using Cluster-Aware Updating:

     Invoke-CauRun -ClusterName <SystemName> -CauPluginName "Microsoft.RollingUpgradePlugin" -CauPluginArguments @{'WuConnected'='true';} -Verbose -EnableFirewallRules -Force
     

   • If using local media (ISO):

     Invoke-CauRun -ClusterName <SystemName> -CauPluginName "Microsoft.RollingUpgradePlugin" -CauPluginArguments @{ 'WuConnected'='false'; 'PathToSetupMedia'='\\some\path\'; 'UpdateClusterFunctionalLevel'='true'; } -Force
     

Step 4: Check the Status of the Update

You can monitor the progress of the update using:

Get-CauRun -ClusterName <SystemName>

Example output:

RunId                   : <Run ID>
RunStartTime            : 10/13/2024 1:35:39 PM
CurrentOrchestrator     : NODE1
NodeStatusNotifications : {
Node      : NODE1
Status    : Waiting
Timestamp : 10/13/2024 1:35:49 PM
}
NodeResults             : {
Node                     : NODE2
Status                   : Succeeded
ErrorRecordData          :
NumberOfSucceededUpdates : 0
NumberOfFailedUpdates    : 0
InstallResults           : Microsoft.ClusterAwareUpdating.UpdateInstallResult[]
}

Post-OS Upgrade Steps

Once the OS upgrade to version 23H2 is complete, you need to perform critical post-upgrade tasks to stabilize the system and enable new features.

1. Upgrade Cluster Functional Level: Upgrade the cluster functional level to enable new capabilities. Note that this step is irreversible:

   Update-ClusterFunctionalLevel
   

2. Upgrade Storage Pool: Identify the storage pool and upgrade it:

   Get-StoragePool
   Update-StoragePool -FriendlyName "S2D on hci-cluster1"
   

3. Upgrade VM Configuration Levels (optional): Stop each VM and update its configuration:

   Update-VMVersion -Name "<VMName>"
   

4. Validate the System: Validate the cluster and check roles, VMs, and live migrations:

   Test-Cluster
   

Install and Enable Network ATC

Network ATC simplifies host networking by automating best practices and ensuring consistency. Follow these steps:

1. Install Network ATC:

   Install-WindowsFeature -Name NetworkATC
   

2. Pause and Prepare a Node: Suspend the node and remove any conflicting configurations, such as existing VMSwitch or NetQos policies:

   Suspend-ClusterNode
   Get-VMSwitch -Name "<VMSwitchName>" | Remove-VMSwitch -Force
   

3. Enable and Configure Intents: Add intents for management, compute, and storage:

   Add-NetIntent -Name MgmtComputeStorage -Management -Compute -Storage -AdapterName pNIC1, pNIC2
   

4. Resume the Node: After configuring the intents, resume the node:

   Resume-ClusterNode
   

Repeat for all nodes in the system.

Validate Solution Upgrade Readiness

Before proceeding with the solution upgrade, validate readiness using the Environment Checker:

1. Install the Environment Checker:

   Install-Module -Name AzStackHci.EnvironmentChecker -AllowClobber
   

2. Run Validation:

   Invoke-AzStackHciUpgradeValidation
   

3. Address Issues: Review and address any blocking or warning issues, such as BitLocker suspension, language settings, or storage space.

Install the Solution Upgrade

Once the environment is validated, install the solution upgrade:

1. Initiate the Upgrade: In the Azure portal, select the upgrade option in the Azure Local resource page and provide the necessary details (e.g., key vault, deployment account, IP range).

2. Monitor Progress: Track the upgrade status under Settings > Deployment in the Azure portal.

3. Verify Upgrade Success: Check the resource group for all expected resources, such as Azure Arc machines, Arc Resource Bridge, and Key Vault.

After the solution upgrade:

• Validate the system’s health.
• Update security settings and enable RDP if needed.
• Create workloads and storage paths for each volume.

Conclusion

Lifecycle Management in Azure Local is a multidimensional topic that involves updates, upgrades, repairs, and ongoing maintenance. By following the guidelines in this article, whether you’re adding new nodes, expanding storage, repairing extensions, or rotating secrets, you can ensure that your environment remains secure, stable, and ready for future needs.

Before carrying out any updates or upgrades, it’s crucial to stay informed about the latest release information and known issues for Azure Local 23H2. Microsoft regularly publishes:

• Azure Local 23H2 release information which highlights new features, improvements, and security updates in each release train.
• Known issues for Azure Local 23H2 which provides a running list of critical issues, workarounds, and advisories to be aware of for each version.

Spending a little time reviewing these resources prior to an update can save you a lot of hassle, minimize downtime, and ensure your system remains in a fully supported state. With solid preparation and the processes outlined here, you’ll be well-equipped to navigate the complexities of Azure Local Lifecycle Management.

Happy upgrading!

Additional Resources and References

Below is a table of all the links referenced in this article, along with a brief description for each: