Azure News - 2026-10-10

At OCP Global Summit 2026, Microsoft will showcase how we’re advancing the infrastructure needed to scale AI, from silicon and systems to the datacenter. Come visit Microsoft at Booth C73 and hear from Microsoft leaders and experts through our keynote and sessions, panels, and workshops across the summit.

Featured Sessions

KEYNOTE

Intelligence Infrastructure: Scaling AI for the Frontier Era

October 13 · 9:15 AM - 9:35 AM · SJCC, Street Level, South Hall

AI infrastructure depends on co-design across silicon, systems, software, and sites. This keynote examines how Microsoft brings those layers together to build scalable, reliable, and efficient infrastructure with security embedded throughout. It explores the path from technology development to a global fleet, how requirements change at scale, and how those lessons shape next-generation platforms. The session also considers sustainable scaling through closed-loop cooling and lower-carbon construction, the role of grid reliability and workforce development in host communities, and the challenges that call for ecosystem innovation.

EXECUTIVE SESSION

Powering the AI Era: Building Sustainable and Efficient Infrastructure at Cloud Scale

October 12 · 2:30 PM - 2:50 PM · SJCC, Concourse Level, 210AE

The explosive growth of generative AI is driving an unprecedented increase in demand for computing infrastructure. As AI workloads scale from training to widespread inference, every layer of the cloud stack is being challenged, from power grids and datacenters to servers, silicon, and software systems. Meeting this demand sustainably requires rethinking how computing infrastructure is designed, operated, and optimized.

Microsoft Breakout Sessions, Panels, and Workshops

Explore Microsoft daily sessions and hear from experts shaping the future of cloud and AI infrastructure.

Tuesday, October 13

Kubernetes was designed around a simple networking model, every pod gets one network interface, and every pod on a node shares the same flat network. For most stateless microservices, this works well as it keeps the model simple and the platform portable.

But a large class of workloads don't fit that shape. Some need network separation, with management, control, and data traffic on separate and isolated networks rather than multiplexed over one interface. Other workloads need predictable performance, a dedicated path for high-throughput or latency-sensitive traffic. Until now, meeting these requirements on Kubernetes meant adding and manually integrating multiple components, or keeping the workload on VMs.

Today we're announcing the public preview of multi-NIC support for pods on AKS, powered by the AKS managed DRANET driver. You can now provision worker nodes with multiple network interfaces connected to different subnets, and attach a dedicated interface directly to a pod using standard upstream Kubernetes APIs, with no Multus, no custom CNI extensions, and no manual interface wiring.

Two ways to use multi-NIC on AKS

Use case 1: A dedicated NIC attached directly to a pod

“As a networking engineer, I want to attach a dedicated network interface directly to a Kubernetes pod so the workload can connect to different networks and route traffic natively, without relying on host-level DPDK/VPP forwarding, custom routing rules, or gateway-based NAT.”
“I also want the secondary interface on my pod to send and receive traffic on a specific, known public IP address, so that downstream partners can allow-list my traffic by IP and no NAT hop adds latency to time-sensitive streams."

This is the pattern where the workload itself expects to see multiple interfaces. The pod asks for a network device the same way it would ask for any other resource, and the interface shows up inside the container alongside the standard CNI interface. No host-level forwarding layer, no NAT gateway in the middle.

What you get

• The DRANET driver is deployed automatically. Once a node pool is configured with secondary interfaces, AKS installs and manages the driver for you; there is no add-on to install and no CRDs to author by hand.
• Declarative, scheduler-aware requests. Interfaces are advertised to Kubernetes as allocatable devices, so a pod that asks for one is placed on a node that can actually provide it, using standard Dynamic Resource Allocation (DRA) objects rather than vendor-specific annotations.
• A predictable interface lifecycle. The driver handles attach, release, and recovery as pods are rescheduled or restarted.
• A public IP address can be assigned directly to a secondary interface, configured at node pool creation time using IP tags. The pod's secondary interface talks to the outside world without going through the load balancer or any address translation layer.
• Both IPv4 and IPv6 addresses are supported on the secondary interface.

What to plan around

• Interfaces are dedicated, not shared. When a NIC is attached to a pod, it moves into that pod's network namespace and is no longer available to the node or to any other pod. Sharing a secondary NIC across pods isn't supported in this release.
• Because of that, pod density on a multi-NIC node pool is bounded by the number of interfaces on the node.
• All interfaces must be attached to the same virtual network as the primary NIC. On a bring-your-own VNet cluster, different subnets within that VNet are supported, but different VNets are not, in this preview. On an AKS-managed VNet, secondary interfaces are currently limited to the same subnet as the primary NIC , since creating a new subnet on a managed VNet isn't supported yet.
• Each pod's secondary interface gets its own dedicated public IP, so plan node pool sizing around how many pods need a tagged public IP of their own. Public IP assignment is configured at node pool creation time through IP tags, so decide your IP range and tagging strategy before you provision the node pool.
• Kubernetes network policy is enforced on the primary interface only. Plan isolation for secondary interfaces at the subnet and NSG layer.

Use Case 2: Multi-NIC nodes with your own NIC management

“As a platform/networking engineer, I want to provision Kubernetes worker nodes with multiple network interfaces connected to different networks and use my own  VPP, DPDK, or any custom routing logic to steer traffic at the node level, without attaching additional interfaces to pods.”  

This is the pattern for teams who already own their dataplane. The node is multi-homed, each interface lands on its own subnet, and your own software decides which traffic goes where. Pods keep a single CNI-assigned interface and stay unaware of the underlying topology.

What you get

• Depending on the selected SKU size, you can configure nodes with multiple network interfaces.
• Full control, and full ownership of the additional nics. AKS provisions the interfaces and keeps them attached to the node. How traffic is steered across them is entirely yours to design, configure, operate, and support. That includes your routing rules, your userspace dataplane, and any troubleshooting of the forwarding path itself.
• The same-VNet constraint applies here too, secondary interfaces can span subnets, but not virtual networks. This applies to bring-your-own VNet clusters. On an AKS-managed VNet, secondary interfaces are restricted to the primary NIC's subnet for now.

Both patterns start from the same node pool configuration. The difference is whether you stop there and manage the interfaces yourself, or let the DRANET hand them to your pods.

How it works

Multi-NIC on AKS starts with a single configuration step: you add secondary network interfaces to a node pool. AKS provisions the NICs on every VM in the pool and installs the DRANET driver alongside them. From there, what happens to those NICs depends on your workload. Reference a resource claim in your pod spec, and DRANET moves an interface into the pod Or Leave them on the node and steer traffic yourself with VPP, DPDK, or custom routing.

Step 1: Provision the node pools with secondary NICs

Secondary interfaces are declared when you create the node pool. The agent pool's network profile carries a secondaryNetworkInterfaces array, and each entry points at a subnet in the same virtual network as the primary NIC:

AKS extends VMSS provisioning to attach those NICs to every VM in the pool. Add more entries for more interfaces, up to what the VM SKU allows. The array length is validated against the SKU at creation time, so a pool with mixed sizes has to be planned around the smallest one.

The node is now multi-homed and the interfaces are visible on the host. If you are building for use case 2, this is where you stop. AKS provisions and keeps the interfaces attached to the node, and how traffic is steered across them, the routing rules, the userspace dataplane, the troubleshooting of the forwarding path, is yours to design, operate, and support.

Step 2: Using AKS managed DRANET driver to hand an interface to the pod

Once a node pool has secondary interfaces configured, and the cluster is using AKS-managed CNI (Azure CNI or Azure CNI Powered by Cilium), AKS automatically installs the DRANET driver as a DaemonSet, packaged and delivered through an AKS-managed Helm chart. There is nothing to deploy yourself and no CRDs to author by hand.
If you're running a BYO CNI cluster, AKS still provisions the secondary NICs, but you're responsible for deploying DRANET (or your own driver of choice) yourself.

The driver does three things:

• Discovers the additional interfaces present on each node.
• Advertises them to Kubernetes as allocatable DRA devices, published in a ResourceSlice.
• Manages the lifecycle: exclusive per-pod allocation, attachment, release, and recovery as pods move or restart.

Inspect the DRANET deamonset, pods, and deviceclass using the following commands:

Now you can create a “ResourceClaimTemplate” that defines how each pod requests a device.

Configure the pod references the template, and the scheduler places it on a node that has a matching NIC available:

A pod referencing a claim comes up with its usual CNI-assigned eth0 plus the allocated secondary interface, and you can confirm both from inside the container.

When DRANET attaches a secondary NIC to a pod, that interface moves into the pod's network namespace and is no longer available on the node or to any other pod. It returns to the node's pool of available devices when the pod terminates. That gives you a clean isolation boundary and deterministic performance, with the trade-off that pod density on a multi-NIC pool is bounded by interface count, not just CPU and memory.

For the full walkthrough including feature flag registration, more examples visit https://aka.ms/aks/multi-nic.

Try it now

Multi-NIC support for pods on AKS is available now in public preview. Start with the documentation for the full walkthrough: provisioning a node pool with secondary interfaces, defining a resource claim, and validating that both interfaces appear inside the pod. 

• Documentation: https://aka.ms/aks/multi-nic
• Upstream project: DRANET on GitHub

Preview is when your input can actually shape the feature. If you hit a limitation or have a use case we haven’t covered, comment below and share your feedback.

For three hours on the morning of October 9, a single H100 in Azure served 1,434 requests to Qwen3.8-27B: 168.7 million input tokens in, 1.27 million output tokens out. At a public list price for this model that day ($0.425 per million input tokens, $0.085 per million cached input tokens and $2.55 per million output tokens), those tokens are worth $24.35. The three hours of VM time cost $4.65 at that region's Spot price; the dashboard below counts them at a round $2 an hour.

This post is the guide I'd have wanted before I started: how to run Qwen3.8-27B on an Azure GPU VM, how to tune it, and how to monitor it end to end, with metrics, logs and traces, using Application Insights and Grafana. The answer to the title comes at the end. The dashboard that measured it comes from the monitoring setup in the middle.

What you're running

Qwen3.8-27B is a 27-billion-parameter dense model from Qwen, released under the Apache 2.0 license. A few of its properties shape how you serve it:

• Hybrid attention. 48 of its 64 layers use Gated DeltaNet linear attention, which keeps a fixed-size state instead of a per-token KV cache. Only the 16 full-attention layers grow with the context, which is why long contexts fit on one GPU.
• A 262,144-token context window, which Qwen documents how to extend to about one million tokens with YaRN.
• Thinking on by default, with a per-request switch to turn it off.
• Multi-token prediction (MTP) layers, which vLLM can use for speculative decoding.
• Vision: it reads images and video as well as text.

The GPU is a Standard_NC40ads_H100_v5: one H100 NVL with 94 GB of memory and 40 vCPUs. Both official checkpoints fit on it:

Use FP8. The H100 runs FP8 natively, each decode step reads fewer bytes, and the memory the smaller weights free up nearly doubles the KV cache. The NVFP4 builds you'll see in vLLM's recipes target Blackwell GPUs, since the H100 has no FP4 tensor cores.

Step 1: Get a Spot H100 VM

Azure Spot Virtual Machines sell spare capacity at a discount, with no SLA, and Azure can evict them with 30 seconds' notice. In October 2026 this size cost $1.55 to $1.84 an hour on Spot in the regions I settled on, which this post rounds to $2 (prices change; check the pricing page). Spot draws on your regional low-priority vCPU quota, 40 vCPUs for this size, not on the H100 family quota, which was zero in my subscription.

Pick the region with data rather than habit. The sample's infra/spot-regions.sh lists each region's eviction-rate bucket and Spot price history, and the current placement score per zone; there's more on that in the Spot section below. Then create the VM with infra/provision-spot-vm.sh:

LOCATION=southeastasia ZONE=2 IDENTITY_RG=rg-llm-telemetry infra/provision-spot-vm.sh

In about five minutes that gives you:

• a resource group, a VNet and a network security group with only the default rules, which deny inbound traffic from the internet
• a NAT gateway for outbound traffic: Docker images, Hugging Face downloads and telemetry
• the VM: Ubuntu 24.04 and no public IP. It uses Trusted Launch with Secure Boot off, because Secure Boot rejects the unsigned NVIDIA kernel modules. It has a 512 GB data disk for weights and images, the Spot Deallocate eviction policy so an eviction keeps the disks, and a managed identity for telemetry

With no public IP and no SSH, you administer the VM through az vm run-command, which the sample's scripts/vm-run.sh wraps: scripts/vm-run.sh <script> runs a local script on the VM as root and prints the output.

Step 2: Prepare the host

scripts/vm-run.sh scripts/bootstrap-host.sh

The bootstrap takes about four minutes. It mounts the data disk at /data, installs the NVIDIA datacenter driver (the 595 branch with open kernel modules, which loads without a reboot), installs Docker Engine with its data root on /data, and registers the NVIDIA Container Toolkit with Docker. At the end, nvidia-smi should report one H100 NVL with 95,830 MiB.

Step 3: Serve the model with vLLM

The stack is a Docker Compose project: a one-shot job that downloads the pinned checkpoint, vLLM, a LiteLLM gateway, and optional overlays for telemetry and a public URL. The sample's scripts/deploy.sh, run with up, copies the folder to the VM through run-command and starts it in the background. On a new VM everything is healthy about 20 minutes later. Most of that is image pulls and the 31 GB download, and later restarts of the model take a few minutes.

The model runs in NVIDIA's NGC vLLM container, release 26.07 (vLLM 0.24). One fix was needed there: tool calls failed with HTTP 500 because the bundled xgrammar 0.2.0 lacks a function vLLM's frontend imports, so the sample's image adds xgrammar 0.2.3. The checkpoint is pinned to a revision and downloaded once to the data disk, so restarts don't depend on Hugging Face.

This is the serve command:

vllm serve /models/Qwen3.8-27B-FP8 \
  --served-model-name qwen3.8-27b \
  --trust-remote-code \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.93 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 16384 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --spec-method mtp --spec-tokens 3 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image":{"count":4,"width":1024,"height":1024},"video":0}'

Step 4: Call it

vLLM serves an OpenAI-compatible API, and so does the LiteLLM gateway in front of it, under one stable alias, local-llm, so clients don't change when the model does:

curl http://127.0.0.1:4000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "local-llm",
  "messages": [{"role": "user", "content": "Explain prefix caching in two sentences."}],
  "temperature": 0.7, "top_p": 0.8, "presence_penalty": 1.5,
  "chat_template_kwargs": {"enable_thinking": false}
}'

Thinking is on unless a request turns it off with chat_template_kwargs. Qwen's model card recommends temperature 1.0, top_p 0.95 and top_k 20 with thinking on, and temperature 0.7, top_p 0.8, top_k 20 and presence_penalty 1.5 with it off. Tool calls work as they do with OpenAI: send tools, and the parser turns the model's tool-call markup into tool_calls.

The sample's smoke test (scripts/vm-run.sh scripts/smoke-test.sh) checks an exact answer, a required tool call and the decode speed. On this VM, one request generates about 166 tokens a second with thinking off.

The VM has no public IP, so the gateway is published through an ngrok tunnel with an IP allow-list. The gateway has no authentication of its own, so that allow-list is the only access control on the public URL. That suited my own tests. For anything more, add a LiteLLM key or an authentication policy at the tunnel.

Step 5: Tune it

Two settings were worth measuring.

How many tokens to draft. Speculative decoding drafts tokens with the MTP layers and verifies them in one pass. On natural-language prompts with 512-token replies:

Three came out on top at both ends. vLLM warns that drafting more than one token reuses the same MTP layer and can lower acceptance, so watch the acceptance rate on the dashboard rather than trusting the setting.

How many requests at once. On synthetic 2,000-token prompts, output grows to 32 concurrent requests and then flattens: 150 tokens a second at 1, 821 at 16, 1,187 at 32 and 1,212 at 48, while each stream slows from 150 to 25 tokens a second. On natural text, where more drafts are accepted, 32 requests reach about 1,550 output tokens a second. That's the measured ceiling: 5.6 million output tokens an hour, worth $14.23 an hour at $2.55 per million.

Step 6: Monitor it: metrics, logs and traces

A self-hosted model has no provider status page. When it's slow, you need to know whether the cause is the queue, the cache, the GPU or the client, and when it's idle, you're still paying for it. Azure Monitor ingests OpenTelemetry natively (generally available since June 2026), so one Application Insights resource can receive all three signals.

Create the destination

1. Create an Application Insights resource with OTLP support set to On. That provisions a Log Analytics workspace for logs and traces, an Azure Monitor workspace that stores metrics as Prometheus data, and the data collection endpoint and rule that route OTLP into them. Ingest OTLP data into Azure Monitor with the OpenTelemetry Collector walks through it.
2. On the resource's Overview page, copy the OTLP connection info: the endpoint URLs for traces, logs and metrics, and the data collection rule ID.
3. Give the VM's managed identity the Monitoring Metrics Publisher role on that data collection rule. The collector signs in with Microsoft Entra ID, so you don't have to store a key on the VM. The sample's infra/telemetry-identities.sh creates one identity per region you run in and assigns the role.

Point a collector at it

One OpenTelemetry Collector (the contrib distribution) runs next to the model and gathers everything:

• Metrics: it scrapes vLLM's and LiteLLM's Prometheus endpoints and NVIDIA's DCGM exporter for the GPU, and reads host and container statistics.
• Traces: LiteLLM sends its spans to the collector over OTLP. Each model call is a span with the token counts, time to first token and finish reason as gen_ai.* attributes.
• Logs: it tails every container's log, plus the host's syslog, where GPU Xid errors and out-of-memory kills appear.

The part that talks to Azure Monitor is short (abridged from the sample's observability/otelcol/config.yaml, collector 0.161):

extensions:
  azure_auth:
    managed_identity:
      client_id: ${env:TELEMETRY_UAMI_CLIENT_ID}
    scopes: [https://monitor.azure.com/.default]
processors:
resource:
attributes:
- {key: microsoft.applicationId, value: "${env:AZMON_APPLICATION_ID}", action: upsert}
exporters:
otlp_http/azuremonitor:
traces_endpoint: ${env:AZMON_TRACES_ENDPOINT}
logs_endpoint: ${env:AZMON_LOGS_ENDPOINT}
metrics_endpoint: ${env:AZMON_METRICS_ENDPOINT}
auth:
authenticator: azure_auth

Here's where each signal lands and how you query it:

Four things that tripped me up

• Set microsoft.applicationId on everything you send. The data collection rule that Application Insights creates routes data on that resource attribute, which is why the processor above sets it on every resource.
• Metrics keep only a few resource attributes. The metrics rule keeps service.*, host.name and cloud.region, and drops the rest, so per-container series merged into one until a transform processor copied the container name onto each data point. The sample does the same for an upstream_model label, so a second model on the same VM shows up as its own series. In logs, fields you want to filter on also belong on the record, not the resource.
• OTLP histograms arrive as native histograms. histogram_quantile takes the bare metric name, with no _bucket suffix and no le label, and OpenTelemetry-native names keep their dots, so you write {__name__="system.cpu.utilization"}.
• Check what your spans carry. LiteLLM records the useful parts of each call (token counts, time to first token, finish reason) without prompts or responses when message capture is off. It still records every tool definition the client declares, though. Decide whether that's acceptable in your Application Insights resource, and choose the fields carefully before you share a dashboard.

Build the dashboard in Grafana

In Azure Managed Grafana, connect the Azure Monitor workspace as a Prometheus data source for the metrics (Connect Grafana to Azure Monitor managed service for Prometheus), and use the built-in Azure Monitor data source for KQL over the logs and traces. If you'd rather not run a Grafana instance, Azure Monitor dashboards with Grafana work in the Azure portal.

You don't have to build the dashboard yourself. The one in these screenshots is in the sample repository as h100-llm.json. To use it:

1. In Grafana, go to Dashboards > New > Import and upload the file.
2. When it asks, enter your Application Insights resource ID.
3. At the top of the dashboard, pick your Azure Monitor workspace's Prometheus data source and your Azure Monitor data source.

It has ten rows: overview, token economics, throughput, the gateway, traffic and latency, cache and scheduling, the GPU, host and containers, the telemetry pipeline itself, and traces and logs. These are the questions I look at first:

Metrics tell you that something is slow. Traces and logs tell you which request and why. The dashboard's last row has both. LLM calls lists one row per model call from the LiteLLM spans, with its time to first token, input tokens, finish reason and trace ID. Model server log shows vLLM's own log lines: the engine stats it prints every ten seconds (throughput, running and waiting requests, KV cache usage, prefix cache hit rate) and any warning.

There's one more step I haven't taken yet: vLLM can export its own spans over OTLP with --otlp-traces-endpoint, and NVIDIA's container already includes the OpenTelemetry SDK. Pointed at the same collector, that would add vLLM's own timing for each request, such as time in the queue, next to the gateway's spans.

Change the dashboard and ask the data with an agent

You don't have to edit dashboard JSON or write every query by hand. Azure Managed Grafana instances in the Azure public cloud include a remote MCP server at https://<grafana-endpoint>/api/azure-mcp (Configure an Azure Managed Grafana Remote MCP server). Connect an MCP client to it, and an AI agent can work with this dashboard and the telemetry behind it. GitHub Copilot in Visual Studio Code signs in with Microsoft Entra ID; Claude Code and other clients can use a Grafana service account token.

• Author and change dashboards. amgmcp_dashboard_inspect reads a dashboard's panels, variables and queries, and amgmcp_dashboard_update creates a dashboard or patches one panel, query or title at a time. A request such as "add a panel for preemptions per minute next to KV cache usage" becomes a targeted patch instead of a JSON edit.
• Query the telemetry. amgmcp_prometheus_query runs PromQL against the Azure Monitor workspace, amgmcp_query_resource_log runs KQL against Application Insights, and amgmcp_query_application_insights_trace fetches a trace. Questions like "which requests waited longest for their first token this morning?" or "how many output tokens did the model generate per VM hour this week?" come back with numbers from the same data the dashboard shows.

The agent acts with the Grafana permissions of the identity it signs in with, so give that identity the role the job needs: Grafana Viewer to ask questions, Grafana Editor to change dashboards. My previous post, Dashboards are for AI agents too, not just humans, shows the other half: how an agent uses a dashboard's queries and descriptions as context.

What the dashboard showed

The traffic in these screenshots is a real agentic workload: long-running sessions that call tools and send the whole conversation back with each request. In the busy window it looked nothing like the benchmark:

• Prompts were long. The median request carried about 120,000 input tokens, and the 95th percentile 186,000.
• Most of it was cached. 88% of input tokens were prefix-cache hits.
• Output was small. There were 133 input tokens for every output token, and 62% of the output tokens were reasoning.
• Concurrency was low. At most 10 requests ran at once, with room for 32.

Output averaged 117 tokens a second, under 8% of the 1,550 ceiling, yet the GPU was busy 77% of the time. The work went into prefill (19.9 million computed input tokens in three hours) and into attention over contexts of 100,000 tokens and more. A single request saw about 34 tokens a second at the median, not 166.

Long contexts also fill the KV cache with only a few requests. Around 08:00 the cache reached 88.6%, and at the same moment the 95th-percentile time to first token jumped to 2.6 minutes: new requests were waiting for memory, not compute. The traces show it request by request, with 200,000-token prompts waiting 40 seconds or more for their first token. About 60 of the window's 64 failed requests were prompts that had outgrown the 262K-token context window, which is where the dashboard's 4.47% error rate comes from.

Measure the shape of your own traffic, not the shape of a benchmark. For this workload the numbers that matter are prefill throughput, the cache hit rate and how many long contexts fit in memory at once.

Spot, honestly

Here is what one VM met in its first week:

Seven evictions in about 40 hours of running, and five attempts to create or restart a VM in UK South that failed for lack of capacity. An eviction deallocates the VM with the model still on its disk, so a restart in place brings the stack back in a few minutes once capacity returns.

I pick regions with two public data sources, which infra/spot-regions.sh combines:

• The SpotResources table in Azure Resource Graph has the eviction-rate bucket and recent Spot price history for each size and region.
• Spot Placement Score rates how likely a Spot request is to succeed right now, per zone, and shows zones where the size is restricted for your subscription. It needs the Compute Recommendations Role.

region               eviction  usd/h    since    min-max
northeurope          0-5       1.55     2026-02  1.55-1.55
uksouth              0-5       1.61     2026-09  1.61-2.06
southeastasia        0-5       1.68     2026-09  1.68-2.28
...
Region         Zone    Score                      QuotaAvailable
southeastasia  1       Low                        True
southeastasia  2       Low                        True
southeastasia  3       RestrictedSkuNotAvailable  True

My rule is the lowest eviction bucket, then the lower price, then a zone that isn't restricted. For this size the placement score was Low in almost every unrestricted zone I checked, all week. Low isn't a refusal, though: VMs scored Low in West US 3, North Europe, UAE North and Southeast Asia all allocated on the first try.

Two things I got wrong at first:

• The eviction rate is per hour. Microsoft Learn describes the eviction rate in the portal as the chance of being evicted within the next hour, based on recent history. At 5% an hour, an eight-hour run has roughly a one-in-three chance of being evicted. Read 0–5% as "plan for evictions", not "evictions are rare".
• Changing zones didn't change much. The first five evictions were all in zone 1, so I moved to other zones. North Europe zone 3 and UAE North zone 2 were evicted too.

Spot fits workloads that can wait for a restart. For steady serving, request quota for regular capacity and drop the Spot flags: the same scripts work, at the regular price.

Does it pay off?

The token economics row at the top of this post turns the dashboard into a running answer. VM hours are the minutes in which the collector could scrape vLLM, so stopped or evicted time doesn't count. The prices are dashboard variables: the VM's hourly price and the per-million prices for input, cached input and output, defaulting to the list prices above. Every stat is an instant query, so after an eviction it reads "Off" instead of holding its last value.

Over the busiest three hours it reads $8.12 of tokens per VM hour, about four times the VM's price. Over the first week it's a different story:

The VMs actually ran about 46 hours that week, counting setup, model loading, benchmarks and the first runs before the telemetry existed. At each region's Spot price that was about $98 of compute, plus a little for disks and the NAT gateway, against $86 of tokens. The first nine and a half hours ran in West US 3 at $3.95 an hour, before I started choosing regions with the data above; the rest averaged $1.67 an hour.

Look at where the value comes from, too. In the busy window, cached input brought in 52% of it ($12.65), computed input 35% ($8.48) and output only 13% ($3.23). Benchmarks measure output tokens per second, but for this workload output was less than a fifth of the value.

So the answer to the title is: it pays off when it's busy. At $8.12 of tokens per busy hour against about $2 of VM, the H100 needs to be busy roughly a quarter of the time to break even, and it can't earn while it's evicted, loading or idle. Self-hosting on an H100 Spot VM makes sense when:

• traffic is steady and its prompts share long prefixes, which is where prefix caching makes input cheap to serve
• you want control over the exact checkpoint, the serving flags and where requests go
• the workload tolerates an occasional restart, or you pay for regular capacity

It makes less sense for bursty traffic that leaves the GPU idle most of the day, or for anything that needs an SLA on a Spot VM.

Try it

• Sample code: wuyi-labs/2610-h100-qwen-38-27b has everything in this post: provisioning, the host bootstrap, the Compose stack, the collector configuration, the smoke test and benchmarks, and spot-regions.sh.
• The dashboard: h100-llm.json, ready to import into Azure Managed Grafana. The sample's README has the import steps.
• The model: Qwen3.8-27B on Hugging Face and vLLM's Qwen3.8-27B recipe.
• Monitoring: Ingest OTLP data into Azure Monitor with the OpenTelemetry Collector, Connect Grafana to Azure Monitor managed service for Prometheus, and, for AI coding agents rather than the model behind them, Monitor AI coding agents with Grafana.
• Agents: the Azure Managed Grafana MCP server, to change dashboards and query telemetry from an AI agent.
• Spot: Azure Spot Virtual Machines, Linux VM pricing and Spot Placement Score.

Then keep it busy.