Author: Arturo Quiroga, Azure AI services Engineer - Senior Partner Solutions Architect — Microsoft

A few days ago I published From Prompt to Production: Building Azure Architecture Diagrams with AI, introducing the open-source Azure Architecture Diagram Builder. One feature got more follow-up questions than any other: the Well-Architected Framework (WAF) validation. Architects from partners and customers — many of whom already use Azure Advisor and the Well-Architected Review — wanted to know exactly what scoring algorithm we use, how it compares to Microsoft's official tools, and whether they should be using all three.

This post is that answer. It's a deep dive into how design-time WAF validation works, how Microsoft's two official WAF assessment algorithms work, and where each fits in the architecture lifecycle.

TL;DR. Microsoft ships two WAF assessment vehicles — the Well-Architected Review (questionnaire, scored from human answers) and the Azure Advisor score (healthy-resources-÷-applicable-resources weighted per subcategory, with Defender Secure Score for Security and cost-weighted math for Cost). Both require either a human filling in a form or live Azure telemetry. Our app runs at design time on a diagram, before anything is deployed, using a hybrid pipeline: a deterministic rule pre-scan followed by an LLM refinement pass. Same five WAF pillars, different lifecycle stage. Complementary, not competitive.

Why design-time validation matters

Every cost overrun, reliability gap, and security incident I've ever debugged was cheaper to fix on a whiteboard than in production. Yet most WAF tooling assumes the architecture already exists — either because there are deployed resources to scan (Advisor) or because someone has built enough of it to answer 60 specific questions about it (WAR).

That leaves a gap. Between "rough sketch" and "deployed resource group" there is no algorithmic WAF feedback loop. That's the gap the Diagram Builder fills.

Microsoft's two official WAF assessment algorithms

Before describing our approach, it's worth being precise about what Microsoft already ships, because the term "WAF assessment algorithm" can mean either of two very different things.

1. Azure Well-Architected Review (WAR) — questionnaire-based

The Well-Architected Review is a free self-assessment hosted on Microsoft Learn.

WAR is human-driven. The algorithm is essentially "how many of the recommended practices have you confirmed you do?" — which is exactly the right algorithm when the assessor is the workload team itself.

2. Azure Advisor Score — telemetry-based

The Advisor score is the closest thing Microsoft ships to a real, deterministic WAF algorithm. It runs continuously over your deployed Azure resources.

The math:

Pillar-specific overrides:

• Security uses Microsoft Defender for Cloud's Secure Score model.
• Cost weights by retail $ cost of healthy resources, plus age-of-recommendation weighting; postponed/dismissed items are removed from the denominator.
• Reliability / Performance / Operational Excellence use the healthy-resources ratio above.

Key terms:

• Healthy resource — a deployed resource with no open Advisor recommendation against it for that pillar.
• Total applicable — resources Advisor was able to evaluate (excludes dismissed/snoozed).

Advisor is the right tool once you're in production. It cannot help you before deployment, because there is nothing to count as "healthy" or "applicable."

The missing stage: design time

Here's the lifecycle, with each tool's domain shaded:

Design / Diagram — Diagram Builder validation runs here.

• Operate / Observe — Azure Advisor runs here continuously.
• Periodic Review — WAR runs here, typically quarterly or at major milestones.

These three stages are sequential and complementary. Our app does not replace Advisor or WAR — it adds a feedback loop earlier in the lifecycle, where corrections are cheapest.

How design-time validation works in the Azure Architecture Diagram Builder

The validator is a two-phase hybrid pipeline: deterministic local rules first, then LLM refinement. The full source lives in three files:

• src/services/architectureValidator.ts — orchestrator and prompt
• src/services/wafPatternDetector.ts — topology + service rule engine
• src/data/wafRules.ts — the rule knowledge base

Phase 1 — Deterministic rule pre-scan (~1 ms, no LLM)

When you click Validate Architecture, the validator runs a fully client-side rule engine against the diagram's services, connections, and groups. There are two kinds of rules:

Architecture-pattern rules

These fire when a topology anti-pattern is detected:

Service-specific rules

Every service in the in the generated Azure Architecture diagram is matched against SERVICE_SPECIFIC_RULES by normalized type — App Service, Functions, AKS, Cosmos DB, SQL Database, Storage, Key Vault, and 22 more.

The knowledge base at a glance

The preliminary score

Each finding has a severity, and severity drives a fixed point deduction from a starting score of 100:

Result is floored at 10 (so even a deliberately bad architecture scores at least 10) and ceilinged at 95 (no findings ≠ perfect — there's always something the model might still catch). This is the deterministic baseline before the LLM ever sees the architecture, and it's what makes the pipeline reproducible.

Phase 2 — LLM contextual refinement

The pre-scan output, the topology, and the optional natural-language description are folded into a focused prompt sent to one of seven Azure OpenAI models (GPT-5.1 through 5.4, GPT-5.x Codex variants, DeepSeek V3.2 Speciale, Grok 4.1 Fast). The system prompt gives the model explicit scoring guardrails:

• Score based on what IS present, not what COULD be added.
• A well-connected architecture with appropriate services should score 60–80.
• Score below 50 only for critical gaps (no auth, no monitoring, single points of failure).
• Findings are improvement suggestions, not reasons to penalize the score severely.

The model returns strict JSON:

{
  "overallScore": 0-100,
  "summary": "2–3 sentence assessment",
  "pillars": [
    {
      "pillar": "Reliability | Security | Cost Optimization | Operational Excellence | Performance Efficiency",
      "score": 0-100,
      "findings": [
        {
          "severity": "critical | high | medium | low",
          "category": "...",
          "issue": "...",
          "recommendation": "...",
          "resources": ["service-name-1", "service-name-2"],
          "source": "rule-based | ai-analysis"
        }
      ]
    }
  ],
  "quickWins": [ /* same shape as findings */ ]
}

Two things to call out:

1. Every finding is tagged rule-based or ai-analysis. That tag is the credibility lever. You can always see what the deterministic engine produced versus what the model contributed on top. If you don't trust the AI layer, you can ignore it entirely — the rule layer still stands.
2. The LLM is given pattern hints, not the entire rule catalog. The prompt stays small and focused, which is roughly 3–5× faster and cheaper than asking the LLM to do everything from scratch.

What the user sees

On every run the modal reports:

• Overall WAF score (0–100)
• Per-pillar score × 5 (0–100 each)
• Severity breakdown — counts of critical / high / medium / low across all findings
• Quick wins — high-impact, low-effort items the model surfaces separately
• Hybrid metadata — local findings count, patterns detected, KB rules used, preliminary score, local elapsed ms
• AI metrics — model used, reasoning effort, prompt/completion/total tokens, elapsed time
• App Insights telemetry — an Architecture_Validated event with model, overall score, finding count, elapsed time

Worked example

Take this prompt, which I've used in demos with partners:

"A multi-region web application: Azure Front Door in front of two App Service instances in West US 2 and East US 2, both reading from an Azure SQL Database with geo-replication, with Application Insights for telemetry. No Entra ID, no Key Vault."

After generation, Validate Architecture runs:

Phase 1 — pre-scan (deterministic), ~1 ms

• Patterns detected: no-identity, no-key-vault
• Findings produced: 8 (1 critical, 1 high, 3 medium, 3 low)
• Preliminary score: 100 − 12 − 7 − (3×3) − (1×3) = 69

Phase 2 — LLM refinement, ~6–9 s depending on model

The model accepts the two pattern hints, validates them in context, and adds three more findings of its own:

Final scores returned by the model:

The Security score is the lowest because two of the highest-severity findings landed there — exactly what a human reviewer would flag first.

Multi-model comparison

Because the deterministic floor is identical across runs, the Validation Comparison view becomes a fair shootout of what each LLM adds on top of the same baseline. The same diagram is scored by all seven models, and the UI surfaces:

• Overall score per model
• Per-pillar score per model
• Severity-count deltas
• Number of ai-analysis findings each model contributed
• Quick wins each model identified

This is genuinely useful for two reasons. First, it shows that LLM scores vary — typically by ±5–10 points on the same architecture — which is exactly why we publish the rule-based vs ai-analysis tag. Second, it lets architects pick the model whose review style matches their own.

How we align with Microsoft's algorithms

Where we deliberately differ — and why

Honest limitations

I'd rather you hear these from me than discover them in production:

• LLM scores drift. ±5–10 points across models on the same diagram is normal. Treat the score as directional, the findings as actionable. The rule-based tag is your anchor.
• No live telemetry. We can't know if your App Service is actually using availability zones — only that you have App Service in the diagram. Advisor will tell you the truth post-deployment.
• Generic ruleset. No specialized workload branches yet (AI/ML, IoT, SAP, SaaS). WAR has those.
• No milestone tracking. Each validation run is independent. Compare runs manually using the Validation Comparison view.
• Rule coverage is finite. 29 services and 73 rules is a strong start but not exhaustive — the LLM layer exists in part to compensate for that gap.

How to use all three together

A lifecycle that actually works:

1. Design — Use the Diagram Builder to sketch the architecture and validate at design time. Iterate until the per-pillar scores look reasonable and the critical/high findings are addressed.
2. Deploy — Generate Bicep from the diagram, deploy, and let Azure Advisor start scoring real resources.
3. Operate — Use Azure Advisor continuously. Use Defender Secure Score for security posture.
4. Periodic review — Run a Core WAR every quarter or at major milestones to capture the things only humans know (business context, tradeoffs, planned debt).

None of these three replace the others. They cover different stages of the same loop.

What's next

A few things on the roadmap I'd love feedback on:

• Milestone tracking so design-time scores can be compared over time the way WAR milestones work.
• Workload-specific rulesets mirroring WAR's branches — starting with AI/ML.
• Direct Advisor handoff — once a diagram is deployed, surface the corresponding Advisor recommendations in the same UI to close the loop.

Try it, fork it, tell me where it's wrong

• Live app: https://aka.ms/diagram-builder
• Source: github.com/Arturo-Quiroga-MSFT/azure-architecture-diagram-builder

Useful references:

• Azure Well-Architected Framework pillars
• Azure Well-Architected Review tool
• Azure Advisor score — calculation
• Use Azure WAF assessments (Advisor)
• Complete an Azure Well-Architected Review assessment

If you're a partner or customer architect who's already living in Advisor and WAR, I'd genuinely value your reaction — does the design-time stage feel like a real gap to you, or are you already covering it some other way? Open an issue on the repo or reply on LinkedIn.

Posted on the Azure Architecture Blog · Comments and issues welcome on the repo.

Audience: Engineering leaders, platform architects, senior developers exploring how to operationalise AI in their teams

Reading time: 8 minutes

Series: Cloud Native Platforms. Build, Run, Evolve. This is Part 3 of 3.

Cloud helped us scale infrastructure.

AI is starting to do the same thing for the work around the code: the planning, the testing, the release communication, the incident triage, the writing that surrounds writing software.

The conversation about AI in software has narrowed too quickly to "Copilot in the editor". The bigger story is happening across the lifecycle. Planning, design, development, testing, release, and operations are all being augmented at once. The platforms that adopt AI well are not the ones with the most usage. They are the ones with the clearest discipline around how it is used.

This post is about that discipline.

AI is changing how we engineer, not how we type

AI is not changing how we write code. It is changing how we engineer software.

Code generation is the surface. Underneath it, AI is reshaping the unit of leverage. The question is no longer how fast a developer can type. It is how well a workflow can be expressed as a reusable engineering asset. Six disciplines determine whether AI moves the needle on outcomes or just adds another tool to the stack.

Figure 1. AI across the SDLC. Each phase has clear AI assist points and clear human-owned validations. The boundary is not negotiable. It is the design.

1. From assistance to augmentation

Early AI tools focused on assisting individual developers. Code suggestions. Autocomplete. Quick refactors. The value was real but bounded by the editor.

The shift now is into structured workflows that span the lifecycle. The unit of leverage is no longer a single suggestion. It is a sequence of actions executed reliably across phases. ("Agentic" later in this post means a system that makes its own next-step decisions inside guardrails. A workflow follows a fixed sequence; an agent chooses the path.)

• Code generation has become baseline, not differentiator
• Workflow generation is where the largest gains live
• Multi-step assistance with explicit human checkpoints
• Context that travels across tools, not just within one

In practice

The pattern that works: start with the single highest-volume writing task on the team (commit messages, code review comments, release notes, postmortem first drafts) and turn the AI assist for that task into a shared workflow rather than each individual's private trick. The cost is one engineer's afternoon documenting the workflow and the eval set. The return is that every engineer on the team inherits the work, and the task that used to consume an engineer's morning every two weeks becomes a background step in the release process. Workflow generation, not faster typing, is where the gains compound across a team.

Code suggestions help one developer. Reusable workflows help the next ten.

2. AI across the SDLC, with guardrails

AI now has a useful role at every phase of delivery. The role is different at each phase, and the guardrails are different too.

The guardrails are not optional decoration. They are the design.

In practice

The pattern that works: stage AI assists for release communication (changelog drafting, customer-facing release notes, internal release announcements) and require a human review before anything goes out. The draft arrives consistently, faster than a human could produce, and easier to compare across releases. The reviewer is not eliminated; the reviewer is moved from author to editor, which is where their judgment actually matters. Teams that adopt this pattern stop missing release-note deadlines and stop publishing inconsistent communication across products.

3. From prompts to reusable assets

Many teams begin with prompt experimentation. Individuals find techniques that work for their tasks. The result is a patchwork of personal practices that do not survive a team change.

The compounding value comes when prompts mature into reusable engineering assets.

Figure 2. The maturity model from prompts to agents. The value compounds at the workflow stage and accelerates at the agent stage. The disciplines that make agents safe are the same ones that made workflows reliable.

The maturity stages, in order of leverage:

• Prompts: ad-hoc, individual, hard to share
• Templates: parameterised prompts versioned with the project
• Workflows: multi-step sequences with clear inputs, outputs, checkpoints
• Agents: autonomous task chains operating within explicit guardrails

The diagram is a maturity ladder, not a graduation. In practice teams operate at all four stages simultaneously for different tasks. A senior engineer may use a one-off prompt to explore a refactor, run a versioned template for commit messages, hand off to a workflow for release notes, and trigger an agent for routine PR triage, all in the same hour. The point of the ladder is not to leave earlier stages behind. It is to know which stage a given task belongs to and to invest accordingly.

In practice

The pattern that works: pick the three prompts your team uses every week, codify them as parameterised templates in the same repository as the application code, and treat them as engineering artefacts (reviewed, versioned, owned). New engineers inherit the team's accumulated practice instead of building their own from scratch. Quality becomes consistent because the variance between individuals shrinks. Investment pays back in weeks, not quarters, and the maturity ladder keeps producing returns as the team moves from templates to workflows to agents.

4. Agentic delivery, with guardrails that survive a security review

The next stage is agentic. AI executes sequences of tasks within a defined scope. The risk is not that the agent will fail. It is that the system around the agent will not catch the failure, and that the failure modes are different in kind from traditional automation. Agents are non-deterministic, they can be manipulated through their inputs, and their actions can have side effects in systems the team does not own.

Five guardrails make agentic delivery safe. The first four are necessary. The fifth is what carries the agent through a security review at a regulated enterprise.

• Identity and scope: the agent runs as a managed identity (or scoped service principal) with the smallest set of permissions that lets it do its job. Permissions are expressed as allowlists, not denylists. Tools fetched at runtime are subject to the same identity boundary as the agent itself.
• Input quarantine: anything the agent reads from a user-controlled source (work item bodies, PR descriptions, customer tickets) is treated as untrusted text. The agent does not execute instructions found in fetched content, and tool calls are validated against an output schema before execution. This is the prompt-injection mitigation, and it is the most common gap in agentic systems shipped today.
• Cost and blast-radius caps: every run has a maximum token budget, a maximum number of tool calls, and a maximum spend. Exceeding any cap aborts the run cleanly. Without caps, scoped credentials are not enough to bound the damage.
• Evaluations and traceability: agents are evaluated against a fixed test set before deployment, and on every prompt or model change. Every action is logged with inputs, outputs, the model and prompt versions used, and the reasoning trace where the model exposes one. Logs are redacted for secrets and personally identifiable information at write time.
• Reversibility taxonomy: actions are categorised by reversibility, not asserted to be reversible in general. A draft write to a private store is reversible. A post to a customer-facing channel is not reversible (deletion does not unsend). A database update may be reversible by a compensating transaction or not at all. Irreversible actions require human approval at the boundary, before they happen, not after. The agent is allowed to draft and stage. The human is the only one who is allowed to make the move that cannot be undone.

In practice

The pattern that works: start with one low-risk agent (release-notes drafter, PR triage assistant) running on read-only inputs, write-only-to-drafts permissions, and a hard cost cap per run. Require explicit human approval at the irreversible step. Wire up an evaluation set on day one, and rerun it on every prompt or model change. Treat regressions as failures, not warnings. The first agent the team ships is rarely the most valuable; it is the rehearsal that establishes the controls every later agent inherits. Teams that skip this rehearsal end up with an agent in production that no one feels safe extending.

Implementation note

An agent without a reversibility taxonomy and a regression eval set is a liability. The discipline is the same one that made workflows reliable: scoped identity, idempotency, traceability, and a clear boundary between machine action and human decision. The YAML below is illustrative, not a runtime contract; it is meant to show the shape of the controls a real agent definition would carry, not the syntax of any specific platform.

5. Where AI still needs human judgment

AI has clear boundaries. The boundaries are not embarrassing. They are the design.

What must stay human-owned:

• Architectural trade-offs and design decisions
• Security validation and threat modelling
• Correctness for business-critical and regulatory paths
• Domain context that has not been written down
• Accountability for outcomes, not just outputs

The goal is collaboration, not replacement. The teams that get the most value from AI are not the ones with the most automation. They are the ones with the clearest sense of where automation ends and judgment begins.

In practice

The pattern that works: name the human-owned items explicitly in the team's working agreement (architecture, security, regulatory correctness, accountability) and audit every AI workflow against that list. When a workflow asks the AI to make a decision in any of those categories, redesign it so the AI prepares the analysis and a human makes the call. Most teams over-trust AI for one of these areas in their first six months and learn the hard way. Naming the boundary up front prevents the lesson from being paid in production. The clarity is the value; the model behind the workflow is interchangeable.

6. Responsible AI is engineering work

The first five disciplines decide whether AI moves the needle. The sixth decides whether the platform can defend the choices it makes with AI. Responsible AI is the engineering practice of building systems whose AI behaviour is fair, transparent, accountable, and safe by design, not by audit after the fact. Treating it as a compliance checkbox at the end of the project is how teams end up shipping AI workflows that fail security review, embarrass the company, or harm users.

Six controls turn responsible AI from a policy into engineering work. These map directly onto the practices Microsoft and the broader industry have converged on, but the names matter less than the practice they enable.

• Fairness in inputs and outputs. The training data, eval set, and prompts are reviewed for systematic bias against any group the system serves. The eval set covers under-represented cases by design, not by accident, and regressions on those cases fail the build.
• Transparency to end users. When a user sees AI-generated content, they are told. When a decision is AI-assisted, the path from input to output is explainable in plain language, not just in a model card buried in documentation.
• Content safety filters. Inputs and outputs pass through safety classifiers (prompt injection, prohibited content, jailbreak patterns) before reaching the model and before reaching the user. Filtering decisions are logged and reviewable.
• Accountability ownership. Every AI workflow has a named owner who is accountable for its outcomes, not just its uptime. The owner has the authority to pause or roll back the workflow when harm is detected.
• Data minimisation and residency. The AI sees only the data it needs to do the task. Personally identifiable information and customer data are scoped, redacted, and kept inside the boundary the customer agreed to. Cross-tenant leakage is treated as a P1 incident, not a feature request.
• Harm evaluation alongside quality evaluation. The eval set measures harm potential (toxicity, hallucination on factual queries, leakage of confidential context) with the same rigour as it measures correctness. Both must pass for a release to ship.

Figure 3. Responsible AI as a set of engineering controls around the AI workflow. The six controls fall into four categories: data discipline (fairness, data minimisation), model discipline (content safety, harm evaluation), deployment discipline (transparency to users), and governance (accountability ownership). All six are necessary; none is sufficient on its own.

In practice

The pattern that works: write the responsible AI plan before the first agent ships, not after the first incident. Pick one workflow that touches user data or generates customer-facing content, and use it as the reference implementation: fairness review on the eval set, content safety filters wrapping the model call, transparency annotation in the UI, redaction of identifying details in logs, harm evals running alongside quality evals on every change, and a named owner with explicit pause authority. The first such workflow takes longer to ship than the unconstrained version. Every workflow after it inherits the controls and ships faster than it would have without them. Teams that defer responsible AI to a future quarter end up retrofitting it under pressure, which is the most expensive way to do it.

A scenario that ties it together

Picture a platform team several months into using Copilot. Adoption is high. Productivity dashboards show gains. But defect rates are not improving and lead time is flat. Leadership asks the obvious question: is AI actually helping, or just feeling like help?

The answer is not to stop using AI. It is to change how AI is measured. Move adoption metrics to the background. Move outcome metrics to the front: defect escape rate, lead time for change, change failure rate, mean time to recovery. In parallel, promote the individual prompts that have proved themselves to shared templates, and the templates to versioned workflows. Retrofit responsible AI controls onto the workflows that shipped first: content safety filters, harm evaluations alongside quality evaluations, transparency annotations on customer-facing output, and a named owner for each workflow.

Six months later, the picture is different. Defect rate improves on the parts of the codebase where reusable workflows were introduced. Onboarding for new engineers is visibly faster. Release notes are consistent across teams. The shift is from celebrating use to tracking outcomes, and once the team measures what matters, the tooling decisions start making themselves.

What teams get wrong

The common pattern is measuring AI by usage, not by outcome. Adoption metrics tell you who tried Copilot. They do not tell you whether defects dropped, lead time improved, or release notes got better.

The fix is not less AI. It is better measurement. The four metrics named in the scenario above (defect escape rate, lead time for change, change failure rate, mean time to recovery) come from the DORA research on software delivery performance and have become a useful default. Two warnings travel with them. First, attribution is hard: an AI workflow rolled out alongside a test refactor and a CI pipeline change cannot claim credit cleanly. Second, baselines matter more than headlines: a single quarter's improvement is not a trend, and a single team's gain is not the platform's gain. Outcome measurement done well needs a baseline window, an attribution discipline, and a kill criterion for workflows that are not paying back. Done poorly, it is just adoption metrics with better names.

There is also the question of cost. AI usage carries a per-run token bill, an evaluation bill on every change, and (for agents) a cost cap that limits damage when something goes wrong. None of these are large compared to the engineering time saved when the workflow works. All of them are visible enough that a finance-aware reader will ask. Track them.

Where to start

The most concrete starter from this post: promote one personal prompt to a shared template. Pick the prompt that gets used most often (commit messages, code reviews, release notes, debugging assist), move it from someone's notes into the repository where the team versions everything else, and watch what changes when the next person on the team runs it. That is the smallest unit of the workflow shift this post argues for, and it is the step where prompts stop being individual practice and start becoming engineering assets.

The shift

The shift is from building systems to building smarter systems:

• AI does not replace engineers. It changes what an engineer's leverage looks like.
• The unit of value is the workflow, not the suggestion.
• The discipline that made platforms operable is the same discipline that makes AI useful.
• Responsible AI is not a compliance step. It is the sixth engineering discipline that lets the other five compound safely.

The series ends here, but the arc is consistent across all three posts. The disciplines that make platforms scale are the same disciplines that make AI useful. Build with discipline. Run with discipline. Evolve with discipline. The tools change. The disciplines do not.

Want to discuss? Where has AI moved the needle most in your delivery, and where has it disappointed you? Drop a comment with patterns you have seen in your environment. Every reply gets read.

Previously in this series:

• Building Cloud Native Platforms That Scale: Patterns That Actually Work. Part 1 covered the design choices that make scale possible.
• Running Cloud Native Platforms: Why Day 2 Decides Everything. Part 2 covered the operational disciplines that decide production outcomes.

This is the third and final post in the series.

Audience: SREs (Site Reliability Engineers), platform engineers, engineering managers running production systems

Reading time: 8 minutes

Series: Cloud Native Platforms. Build, Run, Evolve. This is Part 2 of 3.

Most systems are designed thoughtfully.

Most operations are inherited reactively.

The systems that survive are not the ones built with the most care. They are the ones operated with the most discipline. Production has a way of revealing every shortcut taken during design and every assumption left unverified.

This post is about what it takes to operate a platform once the build is done.

How they are run, not how they are built

Systems are not defined by how they are built. They are defined by how they are run.

A well-designed system that is operated reactively will fail in production. A modestly designed system that is operated with discipline will outperform it. Five operational disciplines decide which side of that line a platform lives on. Each one is engineering work, not a checklist for someone else to handle.

Figure 1. The incident lifecycle as a state machine. The states are not optional steps. They are the contract between the team and the system.

1. Observability is the backbone of reliability

Without observability, every operation becomes a guess. As systems grow, the cost of guessing rises faster than the cost of seeing.

Part 1 of this series argued that observability is a design property: instrumentation contracts, request id propagation, structured logging schemas. Production is where those design choices either pay off or do not. Strong observability in production is a contract that lets any engineer answer three questions in minutes: what failed, why it failed, and what the impact was. The shape of that contract matters more than the tool that implements it. (This three-question framing is community-popularised through the SRE community and writers such as Charity Majors. See Honeycomb's What is Observability for the canonical articulation of the three-pillars and question framing; the substance is older than the framing.)

• Dashboards organised around user journeys, not infrastructure components
• Service level indicators (SLIs: the specific measurements you care about, e.g., success rate, p99 latency) chosen from the user's perspective, not the database's
• Alerts that page only on burn-rate against an SLO (Service Level Objective: the target value of an SLI, e.g., 99.9% of requests complete in under 800ms over a rolling month) using a multi-window strategy. A short window catches fast burns; a long window catches slow drifts. This is what makes SLOs operational rather than decorative.
• Sampling and retention tuned for cost, but never for blind spots
• The distinction between MTTA (mean time to acknowledge: how fast someone notices) and MTTR (mean time to restore: how fast service returns) tracked separately. Conflating them hides whether the team's bottleneck is detection, response, or fix.

In practice

The pattern that works: rebuild the operational view around two or three user journeys (sign-in, place order, view history) rather than per-component charts. Tie alerts to error budget burn rather than raw threshold crossings. Track MTTA and MTTR separately so the team's actual bottleneck (detection, response, or fix) is visible. The investment is rethinking what to measure, not buying a new tool. The return is that incidents stop being discovered by customer complaints first. Teams that make this shift typically find their existing telemetry was sufficient; only the questions being asked of it were wrong.

If a dashboard cannot answer "what is the user experiencing right now", it is not an observability dashboard. It is decoration.

2. Alerts are signals, not notifications

More alerts do not mean better monitoring. In practice, the opposite is true. Once alerts outpace the team's ability to act, important signals start getting missed.

Effective alerting works to a small set of rules:

• Severity that maps to action, not to technical category
• Ownership baked in, never inferred at runtime
• Thresholds tied to user impact, not raw metric values
• Noise treated as a defect, with a regular review cadence
• Suppression and grouping for known multi-alert patterns

In practice

The pattern that works: audit every alert against one test, "what action would I take in the next five minutes if this fires now?" Demote alerts with no answer to dashboards. Remove alerts where the answer is the same as another alert's. Group related alerts so one incident produces one page, not twelve. Most teams discover their alert volume drops by an order of magnitude after a thorough audit, and the alerts that remain start getting trusted again. Trust is the precondition for every other operational practice. Without it, on-call rotations decay into noise filtering and the real signals get missed.

Figure 2. From raw events to pages, in approximate orders of magnitude. The numbers vary by team and workload; what does not vary is that each stage needs to remove one to two orders of magnitude of noise. Teams that page on raw events end up with on-call rotations nobody trusts.

3. Incident response is a practiced muscle

Failures are inevitable. Unstructured response is not.

The teams that recover quickly do not improvise during incidents. They follow a structure that has been practiced when nothing was on fire. The structure is intentionally simple, because incident time is the worst time to negotiate roles.

• Clear roles: incident lead, communications lead, scribe, subject matter expert (the RACI model, Responsible-Accountable-Consulted-Informed, adapted for incident response)
• Defined escalation paths with clear handoff criteria. Escalation means re-paging to a higher tier or specialist, not returning to detection. The lifecycle diagram in Figure 1 makes the distinction explicit.
• Runbooks for the top failure modes, kept short enough to actually be read
• Status communication on a fixed cadence, even when there is nothing new to say. Customer comms and internal comms are tracked separately.
• Blameless postmortems (focus on the system that allowed the failure, not the person who pushed the button) that produce action items the team actually completes
• Game days: scheduled exercises that simulate failure modes (region outage, dependency unavailability, traffic spike) under controlled conditions, so gaps in runbooks are found before incidents do

In practice

The pattern that works: name the incident lead and the comms lead before the first message goes out. Write runbooks short enough to be scannable at 3 AM. Run blameless postmortems with action items that actually get tracked to completion. Schedule game days quarterly so the runbooks are exercised before real incidents. Teams that operate with this structure do not have more engineers; they have engineers who are not single points of failure during recovery. The deepest experts stay the deepest experts, but the platform stops depending on whether they happen to be online.

Implementation note

A short, well-structured runbook outperforms a long, exhaustive one. The goal during an incident is not to think. It is to act on a procedure that has been thought through in calmer times.

Runbook header pattern (keep it scannable in incident time)

title: High latency on order API slo_protected: # this runbook protects two SLOs

• order-completion-success
• order-completion-latency severity: # derived from burn rate, not declared fast_burn: P1 # 14.4x budget burn over 1 hour => page now slow_burn: P2 # 6x budget burn over 6 hours => investigate owner: payments-team indicators: # triggers for evaluation, not severity
• p99 (99th-percentile) latency exceeds the SLO target for 5 min
• error rate exceeds the SLO target for 3 min on order-completion first_actions:
• Open the order-journey dashboard. Confirm impact in business terms.
• Check Service Bus queue depth and dead-letter rate (the most common cause of API latency under load is downstream backpressure)
• Verify Cosmos DB RU/s saturation and partition hotspots
• Inspect the most recent deployment for behavioural changes escalate_if:
• Latency does not recover in 15 min
• Error rate exceeds 5% (fast burn against the SLO)
• Customer reports arrive before our own signals do rollback_path:
• Feature flag "new-order-pipeline" can be disabled per-tenant
• Last known good deployment id is in the release tracker note_on_scaling:

CPU is rarely the cause of latency in this service. Scale only after

confirming the bottleneck is compute, not a downstream dependency or

queue depth. Adding capacity to a saturated downstream amplifies the

incident; it does not resolve it.

The general principle behind that last note travels beyond this runbook: scale-out is the right remediation for compute saturation, not for downstream saturation. When latency rises because a database, queue, or external dependency is saturated, adding capacity in front of the bottleneck moves more requests into the bottleneck and makes the incident worse. This is one of the most common operational mistakes when the dashboard shows red and the on-call instinct says "add more".

4. Release confidence is engineered

Releases get harder as systems grow. The platforms that ship confidently at scale have engineered the path, not learned to fear it.

The patterns that change the math:

• Feature flags that allow change without deploy
• Canary deployments (releasing the new version to a small slice of traffic first, watching error budget burn before continuing) that surface problems on a small slice
• Gradual rollouts with automated rollback triggers
• Database migrations split from application releases
• Release coordination that scales with services, not with team size

In practice

The pattern that works: every change ships behind a feature flag, canary deployments take a small slice of traffic first, and rollback is a one-click step in the pipeline rather than a procedure to be invented during an incident. The cost is the discipline of building rollback paths and exercising them. The return is releases that stop being events. Issues that previously triggered full rollbacks get isolated to a slice and rolled back automatically before they reach most users. The willingness to ship smaller, more frequent changes follows directly from the confidence that bad changes can be undone fast.

Big releases feel safe because they are rare. They are actually risky because every change rides together.

5. Reliability is continuous, not a milestone

Reliability is not achieved through tools alone. It requires continuous refinement, feedback-driven improvement, and a budget that the team can spend on operational work without negotiating each time.

The disciplines that keep systems reliable over years are codified well in the SRE-book framing of service level objectives and error budgets (the canonical reference is the Google SRE Book chapter on Service Level Objectives, with the operational follow-up in the SRE Workbook chapter on alerting on SLOs). The names matter less than the practice they enable.

• SLOs chosen from the user's perspective, with two or three per service rather than ten. More SLOs means none of them shape behaviour.
• Error budgets: the inverse of the SLO, expressing how much unreliability the team is willing to spend in a window. Used up early in the month means slow down on releases. Healthy means feature work keeps moving.
• Multi-window burn-rate alerting turns SLOs from dashboards into pages: short window catches catastrophic failures, long window catches slow drift. Without burn-rate alerting, SLOs are observation, not operation. (The pattern is documented in the SRE Workbook.)
• Reliability work has its own backlog, prioritised against features. Not a wishlist after every incident.
• Regular game days that exercise failure modes (region failover, dependency outage, traffic spike) before they happen for real
• Capacity planning informed by data, not by anxiety

In practice

The pattern that works: define two or three SLOs per service, expressed from the user's perspective. Compute the error budget weekly. When the budget is healthy, ship feature work. When the budget is burning fast, slow down and fix the cause. The conversation about which incidents matter and which can wait becomes possible because there is a shared number to point at. Reliability becomes a quantified property of the platform, not an opinion debated at every retrospective. Teams that adopt this discipline stop having the recurring "how reliable do we need to be?" argument and start having data-grounded trade-off discussions instead.

A scenario that ties it together

A platform was launching a new region. The build had gone well. Day 1 was clean. Two weeks in, latency started creeping up during peak hours. Alerts fired on raw thresholds, but no one could tell which ones to trust. Incident calls turned into long debugging sessions because three different teams owned overlapping pieces of the request path.

The team did not start by buying a new tool. They started by treating operations as engineering work. The dashboard was redesigned around the user journey. Alerts were audited and most were demoted or removed. Roles for incident response were written down. A short runbook covered the top failure modes. Releases were broken into canary slices behind feature flags.

None of this was new. It was discipline applied consistently to work that was previously assumed to be someone else's. The next region launch took half the effort, and the team's mean time to restore on the failures that did happen was measurably lower.

What teams get wrong

The common pattern is treating Day 2 as the cost of Day 1. Teams design beautifully, ship fast, then quietly absorb the operational debt. Dashboards proliferate. Alerts grow louder. Postmortems pile up.

The fix is not more dashboards. It is treating operations as engineering work with the same rigour as feature delivery. Operability is a property the system either has or does not. It is not earned by adding monitoring. It is earned by designing for visibility and operating with discipline.

Where to start

The most concrete starter from this post: an alert audit. List every alert that fires in the next week and apply a single test to each one: "what action would I take in the next five minutes?" Demote the alerts that have no answer. Remove the alerts where the answer is the same as another alert's. The audit takes a morning. The result usually halves alert volume and lifts trust on what remains, which is the precondition for every other operational practice in this post.

The shift

The most important shift in maturity is not technical. It is in stance.

The shift is from shipping software to operating systems:

• Operations is not a phase that follows engineering. It is engineering.
• Reliability is not a milestone reached. It is a discipline practiced.
• Incidents are not interruptions to the work. They are the work.

The teams that internalise this shift run platforms that are smaller, calmer, and more trusted. They do not have fewer incidents because their systems are more advanced. They have fewer incidents because their operational discipline is more consistent. Part 3 of this series argues that the same discipline applies again, in a different domain: the practices that make platforms operable are the practices that make AI useful in delivery.

Want to discuss? What is the one operational practice your team adopted that changed how you sleep at night? Drop a comment with patterns you have seen in your environment. Every reply gets read.

Previously in this series: Building Cloud Native Platforms That Scale: Patterns That Actually Work. The first post covered the design choices that make scale possible.

Next in this series: AI-First Platform Engineering: From Copilot to Agentic Delivery. Cloud helped us scale infrastructure. The next post looks at how AI is now changing how we build and run platforms.

Audience: Cloud architects, platform engineers, engineering leaders making design decisions

Reading time: 8 minutes

Series: Cloud Native Platforms. Build, Run, Evolve. This is Part 1 of 3.

Most engineering teams can build systems.

Few can scale them without rebuilding them.

As platforms grow, complexity does not increase linearly. It multiplies across users, services, tenants, regions, and integrations. The systems that struggle and the systems that scale are rarely separated by which cloud they run on. They are separated by a handful of design choices made early and applied consistently.

This post is about those choices.

The differentiator is not the cloud

Scalable platforms are not built with the right tools. They are built with the right design choices.

Cloud services have closed the gap on infrastructure. The differentiator is no longer which managed service a team picks. It is whether the platform is designed to absorb change, tolerate failure, and support visibility from day one. Five engineering disciplines determine whether a platform scales gracefully or collects technical debt while it grows.

Figure 1. The five disciplines compound into platform scale. Any one neglected becomes the constraint that forces a rewrite later.

1. Flexibility is the foundation of scale

Hard-coded systems work until they do not. The first request to add a tenant, a region, a SKU (a sellable product variant), or a regulatory variant is the moment a rigid design starts to bend. Each subsequent request adds weight.

Scalable platforms move behavior out of code:

• Configuration replaces conditional logic
• Feature flags enable safer, tenant-scoped rollouts
• APIs evolve through versioning, not breaking changes
• Schemas evolve additively. Breaking changes go through versioned contracts with a deprecation window long enough that consumers can migrate without downtime.

In practice

The pattern that works: configuration in a managed store, feature flags with tenant scope, and APIs versioned per consumer contract. Cost is the discipline of treating configuration as code (versioned, reviewed, audited). The return is that releases stop being events and start being routine. A change that previously needed a coordinated deployment can be executed in minutes, gated to a single tenant for verification, and rolled out broadly only after the signal is clean. Most platforms reach this state by retrofit, not by design. Doing it earlier costs less than waiting.

If a change requires a redeploy, it should require a very good reason.

2. Failures are normal. Resilience is a choice.

Distributed systems will fail in unpredictable ways. The real question is not how to prevent failure. It is how the system responds when failure happens.

Resilience is engineered, not inherited from the platform. The patterns that move the needle are well known and consistently applied:

• Idempotent operations (safe to call multiple times with the same result) that make retries safe
• Reliable messaging patterns such as the transaction outbox (writing the message to the same database transaction as the business change, then publishing asynchronously) to avoid lost or duplicated events
• Decoupled services that contain blast radius (the scope of damage when one component fails)
• Timeouts, retries, and circuit breakers (a wrapper around a dependency that stops calling it for a cool-off window after repeated failures) tuned per dependency
• Bulkheads (isolation pools, often a separate compute or queue lane per workload class) that keep noisy neighbours from starving critical paths of resources

In practice

The pattern that works: every write that can be retried carries an idempotency key, every queue consumer is safe to replay, every event published goes through an outbox in the same transactional unit as the business change. When peak load triggers retries, duplicates collapse cleanly instead of producing duplicate orders, double-charged customers, or split-brain state. The contract changes outwards: callers can retry without thinking, queues can be at-least-once instead of exactly-once, and recovery moves from a manual cleanup task to a property of the system. Most teams that adopt this pattern stop seeing certain classes of incident entirely.

Implementation note

An idempotent API is not just a design preference. It changes how the rest of the system can be built. Once writes are safe to repeat, retries become cheap, queues become trustworthy, and recovery becomes automatic.

The naive implementation (read the key, if absent process and save) has a race. Two concurrent requests with the same key both miss the lookup, both call the processor, and both attempt to save. That is the failure mode idempotency exists to prevent. The pattern that survives production is an atomic reserve-then-execute: insert a row keyed by the idempotency key with a unique constraint before doing any work. The first writer wins. Concurrent callers either wait for the original to complete and read its result, or they receive a conflict response.

Task CompleteAsync(
    string idempotencyKey, OrderResult result, CancellationToken ct);

Task<OrderResult> GetCompletedResultAsync(
    string idempotencyKey, CancellationToken ct,
    TimeSpan? maxWait = null);

}

public readonly record struct Reservation( bool IsFirstWriter, string RequestHash);

// Idempotency via atomic reserve-then-execute. // First writer wins; replays return the original result; concurrent // duplicates lose the race and read the winner's outcome (or get 409). public async Task<OrderResult> CreateOrderAsync( Order order, string idempotencyKey, CancellationToken ct) { var requestHash = StableHash(order); // canonical content hash

// Atomic insert: succeeds for the first caller, fails for the rest.
var reserved = await _store.TryReserveAsync(
    idempotencyKey, requestHash, ct);

if (!reserved.IsFirstWriter)
{
    if (reserved.RequestHash != requestHash)
        throw new IdempotencyKeyReusedException();

    // A previous run committed (return its result) or is in-flight
    // (poll with a bounded deadline; 409 if exceeded).
    return await _store.GetCompletedResultAsync(
        idempotencyKey, ct, maxWait: TimeSpan.FromSeconds(5));
}

// We are the first writer. Execute, persist, mark complete.
var result = await _processor.ProcessAsync(order, ct);
await _store.CompleteAsync(idempotencyKey, result, ct);
return result;

}

Three production details matter:

• TTL or compaction on the idempotency record. Without it, the store grows forever. Most teams retain records for the request retry window plus a safety margin (commonly 24 to 72 hours).
• Stable content hash, not the default object hash code. The request hash detects key reuse with a different body, so a client that reuses an idempotency key with a different payload receives IdempotencyKeyReusedException rather than silently getting the wrong result. Canonicalise field ordering, locale, and null handling before hashing.
• Bound the in-flight window explicitly. The genuinely hard case is when the processor succeeded but the store write failed. Production-grade implementations either run the side-effect and the store write in the same transaction (when the processor and store share a database) or use the transaction outbox pattern to bridge them. The poll-with-deadline in GetCompletedResultAsync handles the duplicate-arrives-mid-flight case; the transactional boundary handles everything else.

3. Observability is not optional

Without observability, teams operate blind. As systems grow, the price of guessing rises faster than the price of seeing.

At build time, observability is a design property. The decisions made before the system reaches production are what determine whether it can be operated at all. The dashboards, alerts, and incident practices covered in Part 2 of this series rely on instrumentation choices made here.

The build-time work that pays off in production:

• Request identifiers propagated through every service hop, every queue, every async boundary, so a single user action can be traced end to end
• Structured logging with a consistent schema (event name, correlation id, tenant, severity) rather than free-form strings
• Metrics emitted at the boundaries that matter (every external call, every queue read or write, every database operation), not only at the entry point
• Tracing libraries integrated at the framework or middleware layer so coverage is automatic, not opt-in
• Schemas designed so business signals (orders, sessions, transactions) and system signals (CPU, latency, errors) share the same identifiers and can be correlated later

In practice

The pattern that works: a single request id flowing through every service hop, every queue, every async boundary, propagated automatically at the framework layer rather than per-call. Add one structured logging schema across services (event name, correlation id, tenant, severity), so that a single query joins business events with system events. The investment is hours of upfront framework wiring. The return is that production diagnosis stops being archaeology. Cross-service questions become single dashboards; postmortems shrink from days to hours; and the dashboards in Part 2 actually work because the data underneath is shaped to support them.

4. Delivery practices set the ceiling

Scaling teams requires scaling delivery. Small inefficiencies in pipelines, environments, and release coordination compound into measurable drag.

Delivery maturity that pays off at scale:

• Pipelines as code, reviewed and versioned like application code
• Parallel deployments across services and regions where dependencies allow
• Infrastructure as code with shared modules, not hand-managed environments
• Automated quality gates: tests, security scans, dependency checks
• Trunk-based development (developers commit to a single shared branch many times a day) with short-lived feature branches and progressive delivery. Important caveat: trunk-based works only when test automation and feature flags are already in place. Adopting it before those foundations exist tends to amplify production incidents rather than reduce them.

In practice

The pattern that works: pipelines run in parallel where dependencies allow, infrastructure provisioning is templated rather than per-environment, and quality gates run automatically rather than as discretionary steps. Sequential deployment of a multi-service platform across three environments takes hours; parallelised deployment of the same change takes minutes. The payback is not only release speed. It is the compounding cost reduction of every wait state for every engineer on every release. Teams that treat pipelines as a product feature, not an afterthought, ship more confidently and recover from bad changes faster because the rollback path was exercised, not invented during an incident.

Slow pipelines are not a tooling problem. They are a design problem.

5. Cost discipline is engineering work

Cloud platforms can become expensive quickly when cost is treated as someone else's problem. Cost is a property of the design, not a quarterly review.

The teams that get this right treat cost the same way they treat performance:

• Elastic compute and storage tiers chosen per workload pattern
• Non-production environments with automated scale-down windows (the easiest savings to leave on the table)
• Tagging discipline so cost can be attributed to a service, a feature, a tenant
• Egress and data-tier choices, not compute, dominate cloud bills past a certain scale. Right-size storage tiers (hot vs cool vs archive), eliminate cross-region chatter, and watch egress on the data plane more closely than compute on the request path.
• Budgets and usage alerts wired into the same channels as reliability alerts
• Cost reviews built into design discussions, not deferred to FinOps (Financial Operations: the practice of managing cloud spend as an engineering concern)

In practice

The pattern that works: non-production environments scale down automatically outside business hours, storage tiers match access patterns (hot, cool, archive), and tagging is enforced so every dollar can be attributed to a service or feature. Cost reviews happen at design time, not after the bill arrives. The biggest savings come from data plane decisions, not compute: cross-region egress, oversized storage tiers, and forgotten test environments dominate cloud bills past a certain scale. Treat cost as a first-class non-functional requirement, alongside latency and availability, and the discipline compounds in every design discussion that follows.

A scenario that ties it together

Figure 2. A reference architecture that puts the disciplines into one shape. The request path is decoupled, the data layer is purpose-fit, identity is brokered by managed identity throughout, private endpoints isolate the data tier from public networks, and observability runs as a first-class lane.

Picture a multi-tenant platform at a growth inflection. Onboarding a new tenant takes weeks because tenant-specific behaviour is hard-coded across services. Every release carries risk because there is no way to roll out a change to one tenant without affecting the rest. Incidents linger because logs and metrics live in different tools and nobody can correlate them in production.

Do not start with a rewrite. Start with the smallest set of changes that unlocks the next year of growth: extract configuration out of code, introduce tenant-aware feature flags, wire a unified observability view into the existing services, and parallelise the pipelines. None of these are architectural revolutions. They are design choices applied with discipline, in the order the disciplines compound.

Eighteen months in, onboarding a tenant takes hours instead of weeks. Releases move from monthly events to weekly increments. Incidents are caught earlier and resolved faster. The platform did not get bigger. It got more capable. The five disciplines did the work; the team made the choice to apply them.

What teams get wrong

The common pattern is architecting for the system you have, not the system you are growing into. It looks like progress because the current sprint ships. Pillars get postponed because they feel like overhead.

The cost surfaces later. Each shortcut becomes a constraint. The constraints compound, and three releases later the team is debating a rewrite.

The fix is not premature abstraction. It is small, deliberate investments in flexibility, resilience, observability, delivery, and cost from day one. The discipline is to make these investments before they are urgent.

Where to start when you cannot do everything at once

Five disciplines is a wall, and real teams cannot fund all five at once. The right order depends on whether the platform is being built fresh or already running.

For a system already in production and already in pain, the SRE community's hierarchy of reliability needs gives the most defensible starting order: monitoring and observability first (you cannot fix what you cannot see), then incident response (close the bleeding cleanly), then resilience patterns (idempotency, retries, decoupling) so the bleeding has fewer reasons to start, then flexibility and delivery so safe change can travel at speed. Cost discipline runs alongside throughout, never as the headline.

For a system being built fresh, the order in this post (flexibility, resilience, observability, delivery, cost) reflects the Azure Well-Architected Framework's emphasis on designing for change, failure, and visibility before scaling teams or workloads. Both orders are defensible. What is not defensible is leaving any of the five for later.

The most concrete starter from this post: request id propagation. A single correlation identifier travelling through every service hop, every queue, every async boundary, costs hours up front and pays back every time someone has to debug production for the rest of the platform's life. It is the smallest unit of the observability discipline and the foundation that the dashboards, traces, and incident response in Part 2 all depend on.

The shift

The most important transformation in scaling a platform is not technical. It is mindset.

The shift is from project thinking to platform thinking:

• Build reusable capabilities, not one-off solutions
• Design systems for long-term evolution, not the next release
• Enable other teams, not just deliver for one team

Tools change. Cloud services evolve. The architectural fashions of this year will not be the architectural fashions of the next. What persists is the discipline behind the choices. Scalable systems are not built by tools. They are built by teams that treat design as continuous work. The same discipline shows up again in Part 2 (operating these systems) and Part 3 (using AI to augment that work). The tools change. The disciplines do not.

Want to discuss? What single design choice has paid the most dividends in the platforms you run? Drop a comment with patterns you have seen in your environment. Every reply gets read.

Next in this series: Running Cloud Native Platforms: Why Day 2 Decides Everything. Building is half the journey. The next post looks at what it takes to operate these platforms once they are in production.

Introduction

In a previous blog post, we described how Azure Virtual Network Manager (AVNM) enables central teams to enforce security admin rules across hundreds of virtual networks—bring consistency and governance to complex enterprise environments.

But enforcement at scale introduces a new challenge: deployment confidence. Security admin rules take priority over NSG rules and can span subscriptions and management groups. That makes them powerful—but a single misconfigured rule can disrupt critical traffic across your entire network. Governance teams need a way to understand the real-world impact of a rule before it reaches production—not after.

This is exactly the problem Azure Virtual Network Manager now solves with the Rule Impact Analyzer—a capability that simulates proposed security admin rules against your real network traffic, so you can see exactly what will change, what won't, and deploy with confidence instead of guesswork.

The Challenge: Understanding Rule Impact Before Deployment

As enterprises scale up their use of security admin rules, a visibility gap emerges. Consider a common scenario: a central governance team needs to block high-risk ports across all production virtual networks. The rules are well-intentioned, but the team has no visibility into which existing traffic flows would be affected. Without a way to preview the impact, teams face an uncomfortable tradeoff—move quickly and risk disruption, or slow down manual review across every affected network.

The Rule Impact Analyzer is designed to close this gap—giving teams with a clear, data-driven view of what a rule of change will do before it reaches production.

What Is the Rule Impact Analyzer?

The Rule Impact Analyzer is a joint capability of Azure Virtual Network Manager and Azure Network Watcher. It lets you simulate proposed security admin rules against traffic data derived from virtual network (VNet) flow logs and Traffic Analytics in your environment.

Instead of relying on manual review, the analyzer evaluates proposed rules against observed traffic and classifies each flow:

• Affected — The proposed rule would change the current evaluation outcome for this flow (e.g., traffic that is currently allowed would be blocked).
• Not Affected — The flow would continue as-is; the rule does not apply.
• Indeterminate — The flow cannot be conclusively evaluated (e.g., insufficient traffic data).

This gives governance teams and network administrators a clear, data-driven view of what a rule of change will do—before it reaches production.

Note: The analysis is based on traffic data available through flow logs and Traffic Analytics. Results reflect recorded traffic patterns; traffic that has not yet been observed will not appear in results.

The Customer Journey: From Rule Authoring to Validated Deployment

The Rule Impact Analyzer fits naturally into the lifecycle of security admin rule management:

This workflow lets teams author rules, simulate impact, review results, and refine policies before committing a single change to production. Teams can cycle through simulation as many times as needed.

Key Capabilities

1. Predicted Impact Visibility

See briefly how your proposed security admin rules would affect existing traffic flows. Results are based on Traffic Analytics data, helping teams make informed deployment decisions.

2. Flow-Level Drill-Down

Go beyond summary counts. Inspect specific source and destination paths, see which rule affects each flow, and identify legitimate traffic that would be unintentionally blocked. This makes it easy to pinpoint issues and refine your rules.

3. Configurable Scope

You don't have to analyze everything at once. Target your analysis to specific:

• Rule collections or individual security admin rules
• Network groups or specific virtual networks

This lets you focus on the areas that matter most, whether you're validating a single rule change or assessing a broad policy rollout.

4. Controlled Iteration

Modify your security admin rules, re-run the analysis, and repeat—as many times as you need. Deploy only when the simulated impact matches your intended connectivity outcome.

5. Inbound and Outbound Evaluation

The analyzer evaluates both inbound and outbound traffic directions, giving you full visibility into the rule's impact across your network.

Real-World Scenario: Locking Down Internet-Exposed Management Ports at Scale

Let’s look at a real-world scenario as an example. Your organization runs hundreds of VNets across multiple subscriptions. Over time, different teams have created NSG rules that allow inbound SSH (port 22) and RDP (port 3389) from broad source ranges — some even from 0.0.0.0/0. Your security team mandates: block all inbound management-port access except from trusted bastion subnets.

The challenge? You can't just flip a switch. Blocking the wrong traffic could be risky, and you want to know the impact of applying the security rules.

With Rule Impact Analyzer, you can:

• Define the proposed security admin rule — deny inbound TCP 22/3389 from all sources except your bastion subnet prefix
• Simulate before you commit — see exactly which VNets, subnets, and NICs currently have traffic matching the rule, and which existing NSG rules would be overridden
• Identify conflicts — spot cases where a team's NSG "Allow" rule would be superseded by your new admin-level "Deny," so you can coordinate before deployment
• Deploy with confidence — roll out the rule knowing the blast radius is fully understood, not guessed

Before Rule Impact Analyzer, this required manually auditing NSG rules across every subscription, cross-referencing with resource inventories, and hoping nothing was missed. Now, a single simulation gives you a complete picture in minutes — turning a week-long audit into a self-service workflow.

How It Works: Architecture and Design

Rule Impact Analyzer uses existing Azure networking telemetry and analytics components. It does not require a separate data collection pipeline.

The following diagram provides an interactive version of the architecture:

Step 1: Traffic Analytics as Ground Truth. The analyzer queries your existing VNet flow logs through Traffic Analytics. No new agents, log pipelines, or storage accounts are required.

Step 2: Log Analytics as the Query Engine. Traffic Analytics data resides in your Log Analytics workspace. The Rule Impact Analyzer runs Kusto Query Language (KQL) queries to retrieve the observed flows relevant to your analysis scope.

Step 3: AVNM Rule Evaluation Engine. The retrieved flows are evaluated using AVNM's own enforcement logic—the same priority ordering, allow/deny behavior, and scope resolution used in production. This ensures that what you see in the analyzer matches what would happen when rules are enforced.

Step 4: Results Correlation and Surfacing. Each flow is classified and surfaced in the Azure Portal with drill-down capabilities—from summary impact counts down to individual flow paths and the specific rules affecting them.

What Means for You

• Uses existing infrastructure. If you already have Traffic Analytics enabled, there is nothing new to deploy.
• No data duplication. Queries run in place within your existing Log Analytics workspace, under your existing RBAC and data retention policies.
• Transparent costs. Only standard Log Analytics query costs apply—no hidden charges or separate billing.

Getting Started

You can access Rule Impact Analyzer from two entry points in the Azure Portal:

1. From Azure Virtual Network Manager: Navigate to your security admin configuration → select a rule collection → launch the Rule Impact Analyzer.
2. From Azure Network Watcher: Navigate to Monitoring → Traffic Analytics → Rule Analyzer.

Both paths lead to the same analysis experience, so you can start with whichever tool fits your workflow.

Prerequisites

Before using the Rule Impact Analyzer, ensure the following are in place:

• VNet flow logs are enabled on the virtual networks you want to analyze.
• Traffic Analytics is configured and sends data to a Log Analytics workspace.
• You have the necessary RBAC permissions to access the AVNM security admin configuration and the Log Analytics workspace.

Steps

1. Enable VNet flow logs and Traffic Analytics on your target virtual networks. Learn more about Traffic Analytics.
2. Author or update your security admin rules in Azure Virtual Network Manager. Learn more about AVNM security admin rules.
3. Launch the Rule Impact Analyzer from either portal entry point, configure your scope (rule collections, network groups, or specific VNets), and run the analysis.
4. Review, refine, and deploy. Iterate your rules until the simulated impact matches your intended outcome, then deploy with confidence.

The screenshot below shows the Rule Impact Analyzer in the Azure Portal. After running a simulation, you can see a summary of predicted traffic impact—total paths analyzed, how many are affected or not affected—along with a detailed results table to drill into individual flows and identify which rule impacts each one.

Why It Matters

Outage Prevention

For organizations rolling out network isolation policies at scale, Rule Impact Analyzer acts as a safety net. By simulating rule impact against recorded traffic patterns, teams can catch misconfigurations before they reach production.

Faster Rule Adoption

Without the analyzer, deploying new admin rules often requires lengthy manual review cycles. With self-service impact analysis, governance teams can validate and deploy rules faster—without waiting for manual approval.

Aligning with Behavior

Security policies express intent—what traffic should or shouldn't be allowed. Rule Impact Analyzer validates whether a proposed rule achieves that intent against your observed traffic, closing the loop between policy design and operational behavior.

Conclusion

The AVNM Rule Impact Analyzer closes the gap between policy intent and deployment confidence. Simulating rules against observed traffic—with no additional infrastructure required—governance teams can validate impact before enforcement.

Enforcement without visibility is a risk. Visibility without enforcement is incomplete. This capability brings both together.

We welcome your feedback as you start using this capability. Share your experience through the Azure Portal feedback button or your Microsoft account team.

Learn more:

• Azure Virtual Network Manager
• Azure Network Watcher
• Traffic Analytics
• AVNM Security Admin Rules
• Using Azure Virtual Network Manager to Enhance Network Security

Authors: 

Deepak Bansal, Corporate Vice President and Technical Fellow, Microsoft Azure, Xinyan Zan, Vice President, Ashish Bhargava, Principal Software Development Manager, and Jay Li, Senior Product Manager