Azure News - 2026-05-20

Overview:

A golden image is a prebuilt, approved system template that represents the ideal baseline for deployment. It includes:

• Hardened operating system configuration (e.g., RHEL)
• Preinstalled software and dependencies
• Security patches and updates
• Organizational compliance standards

Architecture:

Golden Image Refresh for VM Scale Sets (VMSS):

Instead of updating instances individually:

• A new image version is published
• The VMSS is updated to reference the new image
• Instances are gradually replaced through a controlled rollout
• New instances (based on updated image) are introduced
• Traffic is gradually shifted to these new instances
• Old instances are decommissioned in phases
• Minimizes service disruption
• Enables safe rollout of updated environments
• Allows real-time validation of new image versions

Virtual Machine Scale Set (VMSS) deployments use a custom image that is baked on top of a Golden Image.
The Golden Image version is pinned in the environment-specific Packer variables (pkrvariables) files.
Refreshing a VMSS Golden Image involves baking a new custom image using an updated Golden Image version and deploying it via the VMSS pipelines.

Image Dependency Flow

1. Golden Image
   • Published and versioned by the Golden Image Team.
   • Source OS image, pinned in pkrvariables per environment.
2. Custom Image
   • Created by the custom image pipeline.
   • Built on top of the pinned Golden Image.
   • Used by VMSS deployments.
3. VMSS Deployment
   • Deploys or updates scale sets using the selected custom image version.

Golden Image Version Management (VMSS)

• Each environment pins the Golden Image version in its respective pkrvariables file.
• Golden Image versions are selected from the same Golden Image Galleries:
  • Dev
  • PPR
  • Prod
• No automatic upgrades occur; changes are explicit and controlled via Git.

VMSS Golden Image Refresh Procedure

1.Select Golden Image Version

• Navigate to the appropriate Golden Image Gallery for the target environment.
• Identify the Golden Image version to be used for the refresh.

2. Update Packer Variables

• Create a feature branch.
• Update the pinned Golden Image version in the environment-specific pkrvariables file.

3.Merge Changes

• Raise a Merge Request (MR) for the updated version.
• After approval, merge the MR into the target branch.

Custom Image Creation

• Trigger the custom image pipeline.
• This pipeline:
  • Uses the updated Golden Image version
  • Bakes a new custom image
  • Publishes a new custom image version for VMSS consumption

VMSS Deployment

Once the custom image is successfully created, deploy it using one of the following approaches:

Option 1: Operational Pipeline

• Use the operational pipeline to deploy the newly created custom image to the VMSS. Operational Pipeline is separate pipeline which will refresh the image.

option 2: Infrastructure Pipeline Update

• Update the infrastructure (Terraform) pipeline code with the new custom image version.
• Run:
  • terraform plan to review VMSS updates
  • terraform apply to roll out the new image

Terraform Behavior

• VMSS instances are updated to use the newly created custom image.
• The same remote Terraform backend is used to preserve state consistency.

Validation and Verification

After deployment:

• Validate VMSS instance health
• Confirm successful instance provisioning
• Verify application and service functionality
• Monitor scale set upgrade status and error metrics

Image Team will provide the golden image and then we need to create custom image.

After retrieval of Custom image used in Infra code.

The Golden image refresh in infra code, requires a activity which is called upgrade and there are 2 kinds of upgrade in VMSS :

Automatic upgrade - VMSS instances will upgrade automatically, and this requires downtime.

All VMSS instances will start upgrading simultaneously and application will be down till VMSS instances is up and running.

Manual upgrade - VMSS instances need to be manually upgraded, and this requires 10 - 15 minutes of degradation.

As part of this Upgrade - we need to manually upgrade VMSS instance one by one and so other instances will be up. There will be no downtime for the application.

Bydefault VMSS will consider automatic upgrade which requires downtime. If we do not require Automatic upgrade then we need to change the setting in provider like below.

provider "azurerm" {
  features {
virtual_machine_scale_set {
      reimage_on_manual_upgrade    = false
      roll_instances_when_required = false
    }
  }

After updating above code in provider.tf as part of manual upgrade then update the terraform code for new golden image. 

• Create a New Image: Start by creating a new golden image with the latest updates and configurations using YAML pipeline. 
• Update Terraform Configuration: Modify your Terraform configuration to reference the new image. This involves updating the source_image_id or image_reference in your azurerm_virtual_machine_scale_set resource to point to the new image version.

Example: 

source_image_id = "/subscriptions/subscriptionid/resourceGroups/rgname/providers/Microsoft.Compute/images/confluence-prd-v-24052450"

data_disks = [
    {
      storage_account_type           = "Premium_LRS"
      caching                        = "ReadWrite"
create_option                  = "FromImage"
      lun                            = 0
      disk_size_gb                   = "500"
      disk_encryption_set_id         = null
      ultra_ssd_disk_iops_read_write = null
      ultra_ssd_disk_mbps_read_write = null
    }
  ]
instances = 3 
automatic_instance_repair = [{
    enabled      = false
    grace_period = "PT30M"
  }]
computer_name_prefix         = "Appname-prd"
overprovision                = false
edge_zone                    = null
health_probe_id              = null
upgrade_mode                 = "Manual"
single_placement_group       = true
secure_boot_enabled          = false

Apply Terraform Configuration: Run terraform apply to apply the updated configuration. This will update the scale set to use the new image.

After the Apply - Upgrade type is Manual then upgrade the VMSS instances one by one to make the service up and running.

Golden Image Refresh for VM Scale Sets (VM)

Scope

• Linux VMs:
  • VMs use RHEL 7.9
  • VMs use RHEL 8.10

For RHEL 7.9 , there is no golden image hence needs to create custom image. To refresh the image, change the image from (example from 1.0 to 1.1)

Resource Changes (VMTRF):

• VM: will be replaced (source_image_id changed)
• OS disk: azapi_update_resource.disk ⇒ replaced
• Data disk attachments: will be replaced
• Network interface: updated in-place
• Disk encryption set: updated in-place
• Role assignments: will be replaced
• VM extension (Custom Script Extension): will be replaced

RHEL 8.10

To refresh the image, change the image from (example from 1.0 to 1.1)

Resource Changes (VM STD):

• VM: will be replaced (source_image_id changed)
• OS disk: azapi_update_resource.disk ⇒ replaced
• Data disk attachments: will be replaced
• Network interface: updated in-place
• Disk encryption set: updated in-place
• Role assignments: will be replaced
• VM extension (Custom Script): will be replaced

We are thrilled to announce that Advanced Container Networking Services (ACNS) for Azure Kubernetes Service (AKS) now delivers two powerful observability features in General Availability: container network metrics filtering and container network log filtering and aggregation. Together, these capabilities set a new standard for Kubernetes network observability, giving you high-fidelity visibility at dramatically lower cost and noise. These capabilities fundamentally redefine how network observability works at scale while delivering up to 97% cost reduction.

Why this is a Milestone?

Most Kubernetes observability solutions face a fundamental tension: collect everything and drown in noise and cost, or sample and miss the signals that matter, with new features of Advanced Container Networking Services that tradeoff has been eliminated.

With this release, Azure becomes the first cloud provider to deliver on-node metrics filtering and flow log aggregation for Kubernetes networking, capabilities now also contributed to the upstream Hubble project, making them available to the broader open-source community.

For AKS customers running Cilium-based clusters, this means:

• Every flow you care about is captured. Everything else is dropped at the source.
• Log volume is compressed by up to 45% through aggregation, without losing security verdicts or error context.
• Costs scale with what you monitor, not with cluster size.

What’s been improved in observability?

This release introduces two capabilities that work together: container network metrics filtering and container network log filtering and aggregation. Both are available on AKS clusters with the Cilium data plane and give you precise controls to keep observability costs predictable while maintaining the visibility you need.

Container Network Metrics Filtering

Container network metrics are generated for all pods by default whenever Advanced Container Networking Services is enabled. With metrics filtering, you now control what gets collected at the point of ingestion, on the node, before anything is scraped or transmitted.

A single ContainerNetworkMetric CRD per cluster defines which metric types (dns, flow, tcp, drop), namespaces, pod labels, and protocols to ingest. It supports both include and exclude filters, so you can maintain broad collection while carving out specific workloads or namespaces. Anything that doesn't match is dropped on the node. Changes reconcile in a few seconds, with no Cilium agent or Prometheus restarts required.

Container Network Log Filtering and Aggregation

Unlike metrics, container network logs are not generated automatically. You start capturing network flows only after applying a ContainerNetworkLog CRD that defines exactly which traffic to capture-by namespace, pod, service, protocol, or verdict. Only matching flows are logged, giving you a precise, targeted view rather than a fire hose.

This is where Azure's first-to-market innovation comes in. Flow log aggregation, now built into Advanced Container Networking Services and contributed upstream to Hubble for the open-source community, groups similar flows into summarized records every 30 seconds. The result is dramatically reduced data volume while preserving security verdicts, service identity, and error context. What previously required custom post-processing pipelines is now built directly into the platform before storage costs are incurred.

Every matched flow log captures: source and destination pods, namespaces, ports, protocols, traffic direction, and policy verdicts.

Logs are stored in a Log Analytics workspace (ContainerNetworkLogs table) with a choice of using the Analytics or Basic tier. Built-in Azure portal dashboards are available for both tiers. Logs can also be exported to external log collectors such as Splunk or Datadog.

First to Market: Azure and the upstream Hubble Contribution

Advanced Container Networking Services built-in filtering and aggregation capabilities were engineered from the ground up to solve real production observability challenges at scale. Rather than keeping this innovation proprietary, Azure contributed the log aggregation and filtering capabilities to the upstream Hubble project, the observability layer of the Cilium ecosystem.

This means:

• AKS customers get a fully managed, Azure-native experience with portal dashboards, Log Analytics integration, and Grafana visualization, out of the box.
• The broader open-source community gains access to the same filtering and aggregation primitives through upstream Hubble.

Azure is the first to ship this capability in a managed Kubernetes service, and the first to give it back to the community.

Key Benefits

💰 Lower observability cost. Metrics filtering drops unwanted data on the node before Prometheus ever scrapes it. Flow log aggregation compresses log data by up to 97% in lab testing. Your cost scales with what you choose to monitor, not with cluster size.

📉 Less noise, more signal. Metrics filtering carves out the namespaces and workloads that matter, so dashboards show only relevant signals. Log filters scope collection to specific pods and verdicts. Engineers start every investigation with data that's already relevant.

⚡ Faster root-cause isolation. Every metric carries source and destination pod context. Targeted flow logs add the forensic detail, which policy, destination, or port is involved. Together, they cut mean time to resolution from hours of guesswork to minutes of structured investigation.

🔒 Full signal, zero gaps. Within the scope you define, every flow is captured and every pattern is preserved. Aggregation compresses volume without losing security verdicts or error context.

Who Benefits

Platform engineers managing multi-tenant clusters can scope data collection per namespace, so each team gets visibility into their own traffic without contributing to a shared cost pool.

SREs can isolate packet drops, TCP resets, or DNS failures to a specific workload in minutes, starting with data that's already scoped to what matters.

Decision-makers evaluating observability spend get predictable, controllable ingestion costs that scale with intent, not infrastructure size.

How to optimize metrics and logs with filtering?

1. Enable Advanced Container Networking Services ( ACNS) on your AKS cluster with the Cilium data plane:

az aks create --enable-acns

Or on an existing cluster:

az aks update --resource-group $RESOURCE_GROUP --name $CLUSTER --enable-acns

2. Apply a ContainerNetworkMetric CRD to filter which metrics are collected on each node. Start by excluding noisy system namespaces, then scope to business-critical workloads.
3. Apply a ContainerNetworkLog CRD to define which flows to capture.
4. Enable Azure Monitor integration with --enable-container-network-logs to send logs to a Log Analytics workspace, or export logs from the node to an external logging system such as Splunk or Datadog.
5. Check your dashboards. Open your cluster in the Azure portal and go to Monitor > Insights > Networking for bytes, drops, DNS errors, and flows. For flow logs, use the built-in Azure portal dashboards available for both Basic and Analytics tiers.

Conclusion

Kubernetes network observability has long meant choosing between visibility and cost. With container network metrics filtering and log filtering and aggregation now GA in Advanced Container Networking Services (ACNS) and contributed to upstream Hubble for the open-source community, that tradeoff is gone.

Azure is first to market with this capability. AKS customers get it fully managed, out of the box, with built-in dashboards with Log Analytics integration. And the broader Cilium ecosystem gets it through upstream Hubble.

High-fidelity visibility. Lower cost. No compromise.

Learn more:

• Container network metrics overview: Container network metrics overview - Azure Kubernetes Service | Microsoft Learn
• Container network logs overview: Container Network Logs Overview - Azure Kubernetes Service | Microsoft Learn
• Configure container network metrics filtering: Configure Container network metrics filtering for Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn
• Set up container network logs: Set up container network logs - Azure Kubernetes Service | Microsoft Learn