The green check that hid a real problem

During a recent customer conversation, I was shown a customer-service refund agent that looked healthy by every traditional measure. Requests completed successfully. The endpoint returned HTTP 200. Latency was within the team's target, and there were no obvious exceptions in the application logs.

Then a customer asked a simple thing:

Please refund the duplicate $500 charge on invoice 83457.

The agent processed the refund and confirmed success. There was only one problem: in the failure case explored in this article, the agent calls the correct tool with the wrong parameter—refunding $5,000 instead of $500.

From the application's perspective, the request succeeded. From the business's perspective, the agent failed.

That moment changed the direction of our discussion. We were no longer asking, "Is the endpoint available?" We were asking:

• Which tool did the agent call, and with what arguments?
• Was the refund amount correct?
• Did the agent honor the approval threshold?
• How many other customers were affected?
• How would we prove that the correction actually worked?

This is the gap that AI observability must close.

Traditional monitoring tells us whether a system is operating. Agent observability helps us determine whether the system is behaving as intended—and whether that behavior produces a trustworthy business outcome.

Below screenshot captures successful but incorrect action by agent. Later in this article, we will build this agent step by step and explore how observability helps us identify the failure.

Observability must extend beyond logs, metrics, and traces

Logs, metrics, and traces remain essential, but an AI application introduces another dimension: the application can be technically healthy and semantically wrong.

I frame this as five layers of AI observability. A refund that returns HTTP 200 can still fail on the layers that matter:

These layers are related, but they are not interchangeable. A response can have low latency and high fluency while still passing the wrong amount to a payment tool. Traditional APM only covers Layer 1.

The Microsoft Foundry observability landscape

For this implementation, Microsoft Foundry was the development and evaluation experience, while Azure Monitor Application Insights became the telemetry store and investigation surface.

The responsibilities were intentionally separated:

• Microsoft Foundry Traces gave the development team an ordered view of the agent run.
• Foundry Evaluation measured quality, safety, and agent behavior.
• The Foundry Monitor experience showed operational and evaluation trends for the deployed agent.
• Application Insights Agent Observability connected runs, models, tools, token usage, latency, and failures.
• Log Analytics supported customer-specific questions with Kusto Query Language.
• Azure Monitor alerts turned important signals into an operational response.

OpenTelemetry provides the connective tissue. It gives us a standard trace model for agent, model, tool, and custom application spans instead of restricting observability to one application framework.

Step 1: Prepare the project

The sample uses Python 3.10 or later, a Microsoft Foundry project, a deployed model, and an Application Insights resource connected to the project. It does not require a preexisting repository or any additional source files. Create a new empty folder and run every snippet in this article in order.

Create and enter the working folder:

Activate the virtual environment on Windows PowerShell:

On macOS or Linux, use:

Install the packages used by the article:

For local development, authenticate with the Azure CLI:

Create a local .env file. Copy the endpoint and deployment name from the Foundry project rather than hard-coding them in source control.

Run Python from foundry-observability-demo. The following code loads .env from that folder and validates every value required by the later snippets

Load and validate the configuration:

Load .env from the directory where Python or Jupyter was started.

if not load_dotenv(dotenv_path=".env"): raise FileNotFoundError( "No .env file was found. Create it in the current working directory." )

Validate the variables required by the article.

required_variables = [ "FOUNDRY_ENDPOINT", "AZURE_TENANT_ID", "APPINSIGHTS_CONNECTION_STRING", ] missing_variables = [ name for name in required_variables if not os.environ.get(name) ] if missing_variables: raise ValueError( "Missing required .env values: " + ", ".join(missing_variables) )

foundry_endpoint = os.environ.get("FOUNDRY_ENDPOINT") tenant_id = os.environ.get("AZURE_TENANT_ID") model_deployment = os.environ.get("FOUNDRY_MODEL", "gpt-4o") app_insights_conn = os.environ.get("APPINSIGHTS_CONNECTION_STRING") log_analytics_workspace_id = os.environ.get("LOG_ANALYTICS_WORKSPACE_ID")

print(f" Foundry endpoint : {foundry_endpoint[:50]}...") print(f"Model deployment : {model_deployment}") print(f"Tenant ID : {tenant_id[:8]}...") print(" Application Insights connection string loaded") print( " Log Analytics workspace ID loaded" if log_analytics_workspace_id else " LOG_ANALYTICS_WORKSPACE_ID is optional until the KQL section" )

Initialize the clients:

credential = AzureCliCredential(tenant_id=tenant_id) project_client = AIProjectClient(endpoint=foundry_endpoint, credential=credential) openai_client = project_client.get_openai_client()

print(" AIProjectClient initialized") print("OpenAI client ready")

The identity running the sample needs permission to use the Foundry project. To query the resulting telemetry, it also needs appropriate access to Application Insights and its Log Analytics workspace. In production, assign access through Microsoft Entra groups and least-privilege roles.

Step 2: Create the refund agent and observe what it actually did

The agent is a function-tool agent named refund-agent-observability-demo. It exposes a single process_refund tool. The instructions are intentionally ambiguous about amount handling to demonstrate how an agent can misinterpret a parameter.

Define the refund tool

process_refund_tool = FunctionTool( name="process_refund", description="Process a refund for a customer invoice. Returns confirmation with refund details.", parameters={ "type": "object", "properties": { "invoice_id": { "type": "string", "description": "The invoice ID to refund" }, "amount": { "type": "number", "description": "The refund amount in dollars" } }, "required": ["invoice_id", "amount"], "additionalProperties": False, }, strict=True, )

tools: list[Tool] = [process_refund_tool]

print(" Refund tool defined") print(f" Tool: process_refund(invoice_id, amount)") print(f" Tool: process_refund(invoice_id, amount)")

Create the refund agent

NOTE: The instructions are intentionally ambiguous about amount handling

to demonstrate how agents can misinterpret parameters

agent = project_client.agents.create_version( agent_name="refund-agent-observability-demo", definition=PromptAgentDefinition( model=model_deployment, instructions="""You are a customer service agent that processes refund requests. When a customer asks for a refund, use the process_refund tool. Extract the invoice ID and amount from the customer's message. Always confirm the refund was processed successfully. """, tools=tools, ), ) print(f" Agent created: {agent.name} (version {agent.version})")

Now send a refund request and inspect exactly what the agent proposed. The "Traditional Monitoring View" stays green while we independently check the tool argument:

user_message = "Please refund the duplicate $500 charge on invoice 83457." print(f" User: {user_message}") print("─" * 60) response = openai_client.responses.create( model=model_deployment, instructions=agent.definition.instructions, tools=[{ "type": "function", "name": "process_refund", "description": "Process a refund for a customer invoice.", "parameters": { "type": "object", "properties": { "invoice_id": {"type": "string", "description": "The invoice ID"}, "amount": {"type": "number", "description": "The refund amount in dollars"} }, "required": ["invoice_id", "amount"], "additionalProperties": False, }, "strict": True, }], input=user_message, )

Inspect what the agent did

print("\n Traditional Monitoring View:") print(f" HTTP Status: 200") print(f" Exception Count: 0") print(f" Response Time: ~2-4s") print(f" Status: SUCCESS") print("\n Agent Output:") for item in response.output: if item.type == "function_call": args = json.loads(item.arguments) print(f"\n Tool Called: {item.name}") print(f" Invoice ID: {args.get('invoice_id')}") print(f" Amount: ${args.get('amount')}") # Check if the amount is correct expected_amount = 500.0 actual_amount = args.get('amount', 0) if actual_amount != expected_amount: print(f"\n PARAMETER ERROR DETECTED!") print(f" Expected: ${expected_amount}") print(f" Actual: ${actual_amount}") print(f" Loss: ${abs(actual_amount - expected_amount)}") else: print(f"\n Parameters correct") elif item.type == "message": print(f" Response: {item.content[0].text if item.content else 'N/A'}")

This is the whole point of observability: the HTTP status, exception count, and latency all look healthy, while the argument check is the only thing that reveals whether the refund amount is right.

The all-green moment:

Step 3: Classify the failure across the five layers

When the amount is wrong, the workshop classifies the same interaction across all five layers—so the "green check" and the real failure sit side by side:

# Visualize the five-layer assessment print("═" * 65) print(" FIVE-LAYER AI OBSERVABILITY ASSESSME

print("═" * 65) print(" FIVE-LAYER AI OBSERVABILITY ASSESSMENT") print("═" * 65) print() layers = [ ("1. Technical", "PASS", "HTTP 200, 0 errors, 2.8s latency"), ("2. Agent Execution", "FAIL", "Wrong parameter: $5000 instead of $500"), ("3. Safety & Policy", "FAIL", "Approval threshold bypassed"), ("4. Quality", "FAIL", "Task executed incorrectly"), ("5. Business", "FAIL", "$4,500 financial loss"), ] for layer, status, detail in layers: print(f" {layer:<22} {status:<10} {detail}") print() print("─" * 65) print(" Traditional APM verdict: ALL GREEN") print(" AI Observability verdict: CRITICAL FAILURE") print("─" * 65)

Traditional monitoring only covers Layer 1. The remaining sections show how to detect, evaluate, monitor, and prevent the other four.

Step 4: Turn on tracing and read the run as a story

Server-side traces become available after Application Insights is connected to the project. Client-side tracing adds visibility into the application logic around the agent call. The workshop enables both with a few lines: it turns on content recording, configures Azure Monitor, and gets a tracer.

from azure.ai.projects.telemetry import AIProjectInstrumentor from azure.monitor.opentelemetry import configure_azure_monitor from opentelemetry import trace

configure_azure_monitor(connection_string=app_insights_conn) AIProjectInstrumentor().instrument( enable_content_recording=True, enable_trace_context_propagation=True, enable_baggage_propagation=False, )

tracer = trace.get_tracer("foundry-observability-blog") print("Azure Monitor configured") print("Foundry GenAI instrumentation enabled") print("Content recording enabled") print("Tracer initialized")

Wrapping an agent interaction in a custom parent span ties the model call and each tool call into one ordered trace. Here the workshop wraps a customer-service session, dispatches the tool the model requested, and returns the result with previous_response_id so the follow-up model call joins the same conversation:

with tracer.start_as_current_span("customer_service_session") as session_span: session_span.set_attribute("session.type", "customer_inquiry") session_span.set_attribute("customer.segment", "premium") # Initial model call response = openai_client.responses.create( model=model_deployment, instructions=agent.definition.instructions, tools=[{ "type": "function", "name": t.name, "description": t.description, "parameters": t.parameters, "strict": True, } for t in tools], input=user_message, ) # Process function calls, return tool output, then continue the conversation input_items = [] for item in response.output: if item.type == "function_call": result = {"status": "processed", "arguments": json.loads(item.arguments)} input_items.append( FunctionCallOutput( type="function_call_output", call_id=item.call_id, output=json.dumps(result), ) ) if input_items: final_response = openai_client.responses.create( model=model_deployment, instructions=agent.definition.instructions, input=input_items, previous_response_id=response.id, ) # Record the trace ID for investigation span_context = session_span.get_span_context() trace_id = format(span_context.trace_id, '032x') print(f" Trace ID: {trace_id}") print(f" View in Application Insights → Transaction Search → {trace_id}")

The resulting trace reads as a sequence, not a pile of logs:

Trace anatomy: 

Graph view of Conversation id: 

 End-to-end investigation in Application Insight

Application Insight- Agents (Preview)

Agents Preview — Models & Tools

Application Insight captures the platform where the agent lives (foundry,copilot studio etc)

Application Insight - failures

Step 5: Ask custom questions with KQL

The portal is an excellent starting point, but customers eventually ask questions specific to their business. The workshop's cookbook targets a Log Analytics workspace and uses the workspace tables (AppDependencies, AppEvents). The legacy aliases (dependencies, customEvents) only resolve in an Application Insights resource query context—use the schema that matches your query scope.

Tool failure rate:

Token consumption over time:

Follow one trace end to end:

// Replace <operation-id> with the actual OperationId let TraceId = "<operation-id>"; AppDependencies | where OperationId == TraceId | project TimeGenerated, Name, DurationMs, Success, ResultCode, OperationId, ParentId, Properties | order by TimeGenerated asc

Correlate human/evaluation feedback with responses:

AppEvents | where Name == "gen_ai.evaluation.result" | extend responseId = tostring(Properties["gen_ai.response.id"]), score = todouble(Properties["gen_ai.evaluation.score.value"]), label = tostring(Properties["gen_ai.evaluation.score.label"]), source = tostring(Properties["microsoft.gen_ai.human_evaluation.source"]) | project TimeGenerated, responseId, score, label, source | order by TimeGenerated desc

Telemetry schemas continue to evolve. Before using a query in production, inspect a representative span and confirm the table and attribute names emitted by the SDK version in use.

Custom KQL:

Step 6: Evaluate the process — did the agent pass the right parameters?

Tracing explains how an action happened. Evaluation measures whether it was correct, consistently. For a refund agent, the most direct check is process evaluation: did it select the right tool and pass the right arguments?

The following example runs builtin.tool_call_accuracy over a set of banking scenarios—including a process_refund case—using the Evals API:

banking_tools = [ { "type": "function", "name": "get_account_balance", "description": "Retrieve the current balance for a customer account.", "parameters": { "type": "object", "properties": { "account_number": {"type": "string", "description": "Account number (e.g., CHK-12345)"} }, }, }, { "type": "function", "name": "process_refund", "description": "Process a refund to a customer account.", "parameters": { "type": "object", "properties": { "invoice_id": {"type": "string"}, "amount": {"type": "number"} }, }, }, ]

Test scenarios with expected tool calls

Note: ToolCallAccuracyEvaluator requires a tool_call_id on every tool_call item.

scenarios = [ { "query": "What's the balance in account CHK-12345?", "tool_definitions": banking_tools, "tool_calls": [{ "type": "tool_call", "tool_call_id": "call_1", "name": "get_account_balance", "arguments": {"account_number": "CHK-12345"} }], }, { "query": "Please refund $75 on invoice INV-9876.", "tool_definitions": banking_tools, "tool_calls": [{ "type": "tool_call", "tool_call_id": "call_3", "name": "process_refund", "arguments": {"invoice_id": "INV-9876", "amount": 75} }], }, ] print(f"{len(scenarios)} test scenarios defined")

import time

Prepare test data — file_content expects an array of {"item": {...}} objects

test_content = [{"item": s} for s in scenarios] testing_criteria = [ { "type": "azure_ai_evaluator", "name": "tool_accuracy", "evaluator_name": "builtin.tool_call_accuracy", "initialization_parameters": {"deployment_name": model_deployment}, "data_mapping": { "query": "{{item.query}}", "tool_definitions": "{{item.tool_definitions}}", "tool_calls": "{{item.tool_calls}}", }, }, ] data_source_config = { "type": "custom", "item_schema": { "type": "object", "properties": { "query": {"type": "string"}, "tool_definitions": {"type": "array"}, "tool_calls": {"type": "array"}, }, "required": ["query", "tool_definitions", "tool_calls"], }, } eval_object = openai_client.evals.create( name="Banking Agent — Tool Call Accuracy", data_source_config=data_source_config, testing_criteria=testing_criteria, ) eval_run = openai_client.evals.runs.create( eval_id=eval_object.id, name="Tool Accuracy Run", data_source={ "type": "jsonl", "source": {"type": "file_content", "content": test_content}, }, ) print(f"Evaluation run started: {eval_run.id}") while eval_run.status not in ["completed", "failed"]: time.sleep(5) eval_run = openai_client.evals.runs.retrieve( run_id=eval_run.id, eval_id=eval_object.id ) print(f" Status: {eval_run.status}") print(f"\n{'' if eval_run.status == 'completed' else '' } Final: {eval_run.status}")

The workshop also runs a quality-and-safety suite (builtin.violence, builtin.fluency, builtin.task_adherence) against a registered agent using evals.create and azure_ai_target_completions. Together, these give both a process signal (right tool, right arguments) and a quality/safety signal.

One honest caveat for publication: in the workshop's saved run, the tool-call records are hand-authored fixtures, not calls captured from a live agent. For a production regression test, capture the agent's actual tool call and its tool_call_id, then evaluate that. For a refund, the exact-amount and approval checks belong in the transaction path as deterministic controls—an LLM judge should not be the only guard.

Evaluation

At first glance, this evaluation run appears healthy: the run completed successfully and achieved 80% tool accuracy. However, the final test case tells a different story. The agent failed the duplicate-refund scenario, receiving a tool-accuracy score of 2 and a result of 0/1. This illustrates why aggregate scores and completion status alone are insufficient—production observability must make individual failures easy to identify and investigate.

A completed evaluation run can still contain a critical failure. The final refund scenario failed tool-call accuracy despite an overall score of 80%.

Step 7: Red-team the agent before it ships

Quality evaluation asks whether expected tasks succeed. Red teaming asks whether deliberate adversarial pressure pushes the agent past its boundaries. The workshop runs the AI Red Teaming Agent in the cloud: it registers a Foundry agent target, generates a prohibited-actions taxonomy, wires up agentic evaluators, and submits a run with attack strategies.

target = AzureAIAgentTarget(name=agent.name, version=agent.version)

Foundry generates the attack-prompt taxonomy from the agent's tools & instructions.

taxonomy = project_client.beta.evaluation_taxonomies.create( agent.name, EvaluationTaxonomy( description="Taxonomy for banking agent red teaming", taxonomy_input=AgentTaxonomyInput( risk_categories=[RiskCategory.PROHIBITED_ACTIONS], target=target, ), ), ) taxonomy_file_id = taxonomy.id

The red team groups one or more runs and wires up the built-in agentic evaluators.

red_team = openai_client.evals.create( name="Red Team — Banking Agent Safety", data_source_config={"type": "azure_ai_source", "scenario": "red_team"}, testing_criteria=[ { "type": "azure_ai_evaluator", "name": "Prohibited Actions", "evaluator_name": "builtin.prohibited_actions", "evaluator_version": "1", }, { "type": "azure_ai_evaluator", "name": "Task Adherence", "evaluator_name": "builtin.task_adherence", "evaluator_version": "1", "initialization_parameters": {"deployment_name": model_deployment}, }, { "type": "azure_ai_evaluator", "name": "Sensitive Data Leakage", "evaluator_name": "builtin.sensitive_data_leakage", "evaluator_version": "1", }, ], )

Create the red-team run — it executes server-side in Foundry.

eval_run = openai_client.evals.runs.create( eval_id=red_team.id, name="Banking Agent Red Team Run", data_source={ "type": "azure_ai_red_team", "item_generation_params": { "type": "red_team_taxonomy", "attack_strategies": ["Flip", "Base64", "IndirectJailbreak"], "num_turns": 5, "source": {"type": "file_id", "id": taxonomy_file_id}, }, "target": target.as_dict(), }, ) print(f"Run created: {eval_run.id} status={eval_run.status}") print(" View in Foundry → Build → Evaluations → Red team")

The strategies (Flip, Base64, IndirectJailbreak) select input transformations and multi-turn depth; they test whether the agent resists manipulation, stays on task, and does not leak data. Two things matter for an honest write-up:

• Review the taxonomy before scanning. Confirm the generated prohibited behaviors reflect your policy. Run red teaming against an isolated target with synthetic accounts and no irreversible side effects.
• Distinguish a security result from an infrastructure failure. A run that ends in failed is incomplete coverage, not a passing scan. Capture the run status honestly, and only present an attack-success-rate scorecard from a completed run.

Red-team run: Capture the run status and, when the run completes, the scorecard in Foundry → Build → Evaluations → Red team. If the run failed, caption it as incomplete coverage rather than a pass.

Step 8: Prove the fix and move to production monitoring

For the sample, the correction is a tightened prompt plus a deterministic amount/approval check in the transaction path—so the agent is not expected to reason its way out of receiving or emitting the wrong amount. After a fix, compare the two versions across the same evaluation set:

The operational success rate does not change. The business outcome does.

Once you know how to detect the failure, the next question is whether it is happening elsewhere. The Foundry Monitor experience brings operational metrics, evaluation results, and red-team results together for the selected agent; Application Insights adds deeper investigation across runs, models, tools, tokens, and errors. For this refund agent, I would monitor at least:

• Run success rate and model or tool errors
• P50 and P95 end-to-end latency
• Input and output token consumption
• Refund-amount and approval-threshold pass rate
• Tool-call-accuracy distribution
• Human escalation and customer correction rate
• Safety and adversarial-testing findings

An alert should lead to an operational action:

A KQL query like the tool-failure-rate example can become the signal for an Azure Monitor log alert. Configure the rule to trigger when the query returns results in the evaluation window, and route the notification through the customer's approved action group.

Continuous evaluation is the bridge between deployment and learning

A one-time evaluation protects one release. It does not protect the agent indefinitely. Production traffic changes, tools change, and models and prompts are revised. This makes observability a continuous engineering loop:

1. Observe production behavior.
2. Diagnose representative traces.
3. Evaluate the failure mode.
4. Add that failure to the regression dataset.
5. Correct the prompt, tool, policy, or model configuration.
6. Compare the candidate with the approved baseline.
7. Promote only when the required quality, safety, operational, and business gates pass.

Microsoft Foundry supports recurring and continuous evaluation. When using live production traces, sample intentionally—random sampling gives coverage, while targeted or intelligent sampling can prioritize unusual, high-risk, failed, expensive, or low-quality interactions. Evaluation calls and telemetry have cost, privacy, and retention implications, so the sampling strategy should reflect business risk.

Security, privacy, and cost are part of observability design

Tracing can capture prompts, model outputs, tool arguments, and tool results. That visibility is valuable, but it creates responsibility:

• Keep message-content recording disabled unless there is an approved need.
• Never place credentials, tokens, or secrets in prompts or span attributes.
• Avoid storing personal data when a pseudonymous transaction identifier is sufficient.
• Redact or minimize sensitive content before telemetry is emitted.
• Apply Azure RBAC to Application Insights and Log Analytics; set retention by environment and data classification.
• Monitor ingestion and evaluation cost, not only model-token cost.
• Validate preview capabilities against production requirements.

There is also a practical instrumentation lesson: do not add every available value to every span. Capture the dimensions needed to investigate reliability, behavior, quality, security, cost, and business outcomes. More telemetry is not automatically better telemetry.

What the customer gained

The most valuable outcome was not another dashboard. It was a shared way for developers, platform engineers, finance/policy owners, security teams, and business stakeholders to discuss the same agent run with evidence.

The developer could see the exact execution path. The platform team could see latency, errors, and token consumption. The policy owner could see the refund amount and whether the approval threshold held. The security team could verify how sensitive telemetry was handled. The business sponsor could see whether customers were refunded correctly.

The original request had a green check before the investigation, and it had a green check afterward. The difference was that after the fix, we could prove the refund amount was correct.

Final takeaway

An agent should not be considered healthy merely because it responds. A production-ready observability practice should be able to answer:

• Did the agent complete the request?
• What model, tool, and path did it use, and with what arguments?
• Was its action relevant, grounded, safe, and correct?
• Did it comply with the approved business process?
• Can the team detect a regression before it affects more users?
• Can the team prove that the remediation improved the outcome?

In the agentic era, observability is not the dashboard at the end of deployment. It is the evidence system connecting design decisions, production behavior, governance controls, and measurable business value.

References

• Observability in generative AI — Microsoft Foundry
• Set up tracing in Microsoft Foundry
• Add client-side tracing to Foundry agents
• Monitor agents with the Agent Monitoring Dashboard
• Monitor AI agents with Application Insights
• Run evaluations from the Microsoft Foundry portal
• Run AI Red Teaming Agent in the cloud
• Azure AI Evaluation client library for Python

Contributors:

This article is maintained by Microsoft. It was originally written by the following contributors.

Gaurav Bhardwaj | Senior Cloud Solution Architect

A few days ago I opened Azure SRE Agent and typed four prompts. None of them was careful:

find grafana dashboard for: GitHub Copilot

run the panels queries. list top 3 very long sessions in the past 3 days

what session 1 was doing?

what was session 2 doing?

No KQL. No table names. I didn't say what a "session" is, what "long" means, or which Application Insights resource my GitHub Copilot telemetry lands in.

Ten minutes later I had this:

and two diagnoses I would not have gotten from the dashboard alone:

• Session 1 (142.7 min) was a GitHub Copilot CLI session building a Grafana dashboard: 291 model calls (233 of them on gpt-6-astra, P50 17 s), 365 tool calls — 162 of them KQL queries through the Azure Managed Grafana MCP endpoint — 3 dashboard writes, and 2 failures. Verdict: "long, not stuck." The cost driver was a ~147:1 input-to-output token ratio: query results and dashboard JSON accumulating in context and being re-sent on every turn.
• Session 2 (47.9 min) was a wait loop: 23 rounds of cost_analysis → bash sleep (60, 120 or 180 s) → retry. The sleeps account for ~44 of the 47.9 minutes. Verdict: "latency-long, not work-long" — and a pattern worth flagging.

On the dashboard these two look the same: a long bar in Agent Run Duration. Underneath they are completely different stories.

The agent didn't know any of that when I started. The dashboard did. That is what this post is about.

First principles

Start from what a dashboard is. Every panel is a query someone wrote because the answer mattered, pointed at the right resource, titled with the question it answers, and annotated with the ways it can mislead. The variables pin it to a subscription, a resource group, an Application Insights component. The tags say what it covers. And it tends to stay correct, because people look at it every day and complain when it isn't. A dashboard is a team's understanding of a system, written down in a form that runs.

That is why a person on call can open one and get insight without asking anyone. It is also the whole argument for handing one to an agent. The insight was never in the pixels; it is in the queries, scopes, titles and descriptions underneath them, and an agent can read those directly.

A dashboard is a map for the agent. It shows where the data lives, which questions are worth asking, how each one is asked, and where the traps are. The agent didn't need a separate context file, a hand-written playbook, or a tour of the schema. Most of what a human needs to reason about the system is already there, and so is most of what the agent needs. The map is already drawn. Here is what the agent found on it.

What the agent did

SRE Agent reached my Grafana instance through the Azure Managed Grafana MCP endpoint — instances in the Azure public cloud have one at https://<grafana-endpoint>/api/azure-mcp — attached as an MCP connector. SRE Agent has a built-in connector for exactly this now; more on that below. The tool trace for the first two prompts is short:

1. amgmcp_dashboard_search — "GitHub Copilot" → one hit: uid GitHubCopilot, tags github-copilot, opentelemetry, application-insights.
2. amgmcp_dashboard_inspect — the panel list, the template variables with their current values, the time range, and every panel's KQL with the variables substituted.
3. amgmcp_datasource_list — the UID of the Azure Monitor data source.
4. amgmcp_query_resource_log — a query the agent composed, run against the Application Insights resource the dashboard points at.

Each "what was session N doing?" was two more query_resource_log calls.

Step 2 is where the interesting part happens, so I connected the same MCP endpoint to my own coding agent and ran the inspect call myself to see exactly what comes back.

Summary mode (no arguments) returns the shape of the dashboard: 21 panels with ids, titles and types; six template variables with their current values; the default time range; and a nextSteps map that tells the agent how to go deeper.

Panel-queries mode returns, for every panel, the query, the data source, and the resource scope, with the template variables resolved on request. This is panel 204, the one that makes long runs visible:

The agent still has to do the last mile. The query tool sends no time window and doesn't expand Grafana macros, so the agent swaps $__interval for a literal bin size and adds its own | where timestamp > ago(3d). But the tool description says exactly that. The agent isn't guessing.

What's on the map

Look at what the dashboard put in that response, and what the agent therefore did not have to figure out.

Where the data is. A single Azure Monitor data source can query any resource its identity is allowed to read, and the scope appears nowhere in the KQL. The dashboard's variables resolved it to one Application Insights component. Without that, the first ten minutes of an investigation often go to "which workspace?"

Which table, which attributes. dependencies, filtered by cloud_RoleName in ('copilot-chat', 'github-copilot') — GitHub Copilot in Visual Studio Code and GitHub Copilot CLI. The source variable is that mapping. Spans are classified by gen_ai.operation.name (chat, execute_tool, invoke_agent); tools by gen_ai.tool.name; tokens by gen_ai.usage.input_tokens and output_tokens. A session is copilot_chat.chat_session_id from VS Code and gen_ai.conversation.id from the CLI, coalesced, because the two clients don't agree.

The pitfalls. This is the part I care about most. Three of the panel descriptions:

Time to first token per model (P50 / P90) for measured requests. VS Code reports copilot_chat.time_to_first_token in ms; Copilot CLI reports gen_ai.response.time_to_first_chunk in seconds (converted to ms here).

Copilot CLI tool calls that cluster at exactly 60 s, 120 s or 180 s are client-side timeouts, which the CLI reports as successful calls.

Agent-run roll-up spans (invoke_agent), which repeat their child requests, are excluded.

Each of those is a mistake I have already made once. Milliseconds versus seconds. Timeouts that look like successes. Roll-up spans that double-count tokens when you sum naively. None of it is derivable from the schema; it is tribal knowledge, and it is sitting in the dashboard JSON where an agent can read it with jsonPath: "$.panels[*].description".

The questions. The 21 panel titles are the questions an experienced on-call engineer asks about this system: Time to First Token by Model. Tool Latency by Tool. Tool Call Failures by Tool. LLM Call Outcomes Over Time. Telemetry Freshness — the last one so you can tell "no problem" from "no data". An agent reading those titles doesn't have to invent a triage plan. It has one.

Off the edge of the map

Notice what the dashboard does not have: a per-session view. There is no "longest sessions" panel; I would have to add one. The agent composed it anyway, from the dashboard's vocabulary:

Every identifier in that query came from a panel. The agent walked off the edge of the map, but it took the map's coordinate system along.

Compare that with what a human does next. The dashboard's Recent Operations table links every run to Trace Visualization, and for one slow tool call that is exactly the right tool. This is session 1's longest run in that view: a single invoke_agent span, 1 hour 18 minutes, 476 spans underneath it.

That view is hard to read end to end. You scroll, sample a few of the long red bars, and move on. The trace is the right shape for "why was this call slow?" and the wrong shape for "what was this run doing?"

The agent never looked at this picture. Its per-session stories came from aggregation instead: execute_tool spans grouped by gen_ai.tool.name, chat spans by gen_ai.response.model, P50s and first/last timestamps per group, and the phases fall out in seconds. Same rows, different reader, different question.

With content capture enabled, the spans even carry gen_ai.tool.call.arguments — which is how "bash, 120 s" became "sleep 120 — Wait before retrying throttled query", and how the agent could say backoff, not hang.

That last step matters more than it looks. The description above says calls clustering at exactly 60/120/180 s are usually client-side timeouts. In session 2 they were the other thing — deliberate sleeps — and only the captured arguments could settle it. A description tells an agent what to check; the telemetry lets it check.

Don't write dashboards for agents

The tempting takeaway is a new checklist: agents read dashboards now, so write dashboards with agents in mind. I think that is backwards. Nothing in the GitHub Copilot dashboard was written for an agent. The titles are questions because the person on call needs to know what a panel answers without reading its query. The descriptions carry the pitfalls because I hit each one and didn't want the next person to. The variables scope the queries because nobody remembers resource IDs. The freshness panel exists because a human can't tell "no problem" from "no data" either. An agent can use every one of those.

Same principle, other direction. What matters to the person on call is what matters to the agent, and the agent is smart enough to work out the rest: the time window, the bin size, a query the dashboard doesn't have yet. If a dashboard is enough for the human, it is enough for the agent. If it isn't enough for the agent, it probably wasn't enough for the next person on call either, and the fix is the same: fix the dashboard, and both readers get it.

Which brings me to the thing I find genuinely new here. A dashboard used to be a terminal artifact — the end of the pipeline, something you looked at. It is now also an input: a compact, versioned, access-controlled, human-reviewed description of how to reason about a system, in a format agents can execute. It runs the other way as well: session 1, the 142-minute one, was a coding agent building a dashboard through the same endpoint (18 inspects, 162 queries, 3 writes), and the longest-sessions query above could become a panel the same way, so the humans get it next time. The same JSON serves both readers, and you only maintain it once.

Connecting SRE Agent to Grafana

I expected the plumbing to be the boring ten minutes: find the endpoint, mint a token, paste both into a connector form. It wasn't, because SRE Agent has a native Azure Managed Grafana connector. Search for "grafana" under Connectors, pick the card, point it at your instance, test the connection, choose the tools. The agent talks to the instance's MCP endpoint with its own managed identity, so you don't have to create or rotate a token.

Try it

• Pipeline and dashboards: Monitor AI coding agents with Grafana — the OpenTelemetry Collector, the VS Code settings that turn on GitHub Copilot telemetry, and dashboards for GitHub Copilot, Claude Code, Codex, OpenClaw, OpenCode and Gemini CLI.
• The dashboard: GitHub Copilot on Grafana.com, ID 25053, ready to import into an Azure Managed Grafana instance.
• The agent: in SRE Agent, add the Azure Managed Grafana connector as above. For anything else — VS Code with GitHub Copilot, Claude Code, your own client — point it at https://<grafana-endpoint>/api/azure-mcp; see Configure an Azure Managed Grafana Remote MCP server.

Then ask it something lazy.

Last week we announced the general availability of TLS/SSL certificate support and end-to-end TLS encryption for Azure Functions Flex Consumption.

Flex Consumption is the recommended serverless hosting plan for Azure Functions. It combines event-driven scale-to-zero and execution-based billing with fast scale-out, configurable instance memory, always-ready instances, and virtual network integration. With this release, customers can now apply a more complete transport-security model to HTTP-based serverless applications without moving to dedicated compute.

The new capabilities include:

• Site-scoped certificates, with up to three private certificates and three public certificates for each function app.
• Multiple certificate sources, including direct upload, Azure Key Vault, App Service Certificates, and free App Service Managed Certificates for supported custom domains.
• Per-certificate access from application code, replacing the broad WEBSITE_LOAD_CERTIFICATES model in Flex Consumption.
• End-to-end TLS encryption between the Azure platform front ends and the workers that run your functions.

Together, these features support common enterprise scenarios such as custom HTTPS endpoints, outbound client-certificate authentication, mutual TLS (mTLS), certificate-based cryptographic operations, and stricter encryption-in-transit requirements.

What end-to-end TLS means for a function app

It helps to separate the different places where TLS and certificates participate in an application request.

First, a client connects to your function app over HTTPS. For the default *.azurewebsites.net hostname, Azure provides and manages the certificate automatically. For a custom hostname such as api.contoso.com, you map the domain to the function app and bind an appropriate server certificate.

Second, Azure Functions receives the request at an Azure platform front end and routes it to a worker that runs your code. HTTPS has always protected the client-to-front-end connection. When end-to-end TLS is enabled, the front-end-to-worker connection is also encrypted. This removes the unencrypted internal hop that otherwise exists after TLS termination at the front end.

Third, your function code might call another service that requires a certificate. For example, a partner API that uses mTLS. A site-scoped private certificate can be exposed to the function runtime and loaded from the Linux file system for that outbound connection.

These controls are complementary:

End-to-end TLS does not replace a custom-domain certificate, and a certificate uploaded for use by code does not automatically configure mTLS. Each control protects a different boundary.

A certificate model designed for Flex Consumption

Traditional App Service certificate resources are scoped to a webspace, a deployment unit associated with an App Service plan's resource group, region, and operating system. Apps in the same webspace can share access to those certificates.

Flex Consumption apps don't share the same plan, so we have introduced a new site-scoped model. The certificate is a child resource of one function app and isn't shared with other apps. This tighter boundary makes certificate ownership easier to reason about, reduces unintended sharing, and aligns well with function apps that are deployed and governed independently like Flex Consumption.

Each Flex Consumption function app supports:

The three-certificate limit includes App Service Managed Certificates, App Service Certificates, Key Vault imports, and directly uploaded PFX files. Plan certificate usage before assigning a separate certificate to every hostname. A certificate with appropriate subject alternative names can be useful when one certificate must cover multiple hostnames.

Choose the right certificate source

Flex Consumption supports several certificate-management approaches:

App Service Managed Certificate. This is the simplest choice for a supported public custom domain. The domain-validated certificate is free, automatically issued, bound, and renewed by App Service. Managed certificates aren't exportable and don't support wildcard domains, private DNS, or use as an outbound client certificate. Avoid certificate pinning because the issuer, public key, and private key can change during renewal.

App Service Certificate. This paid option provides Azure-managed lifecycle capabilities with support for scenarios such as wildcard domains and export.

Azure Key Vault import. This is generally the preferred option when an organization centrally governs certificate issuance, access, and rotation. Use a managed identity and grant the Key Vault Certificate User role. When the certificate is renewed in Key Vault, the platform synchronizes the new version to the function app within 24 hours. If the vault blocks public access, enable the trusted Microsoft services bypass so the Azure Functions platform can retrieve the certificate.

Direct private certificate upload. Upload a password-protected PFX that includes the complete certificate chain, including intermediate and root certificates. For a custom-domain binding, the certificate must include the server-authentication extended key usage and be signed by a trusted certificate authority. Elliptic Curve Cryptography certificates are supported when uploaded as PFX.

Direct public certificate upload. Upload a CER when function code needs a public certificate. Public certificates aren't used to secure custom domains.

Configure the security foundation with infrastructure as code

The following Bicep excerpt shows the relevant settings on the function app. Add these properties to a complete Flex Consumption deployment that also defines the FC1 plan, deployment storage, runtime, identity, and monitoring resources.

The settings have distinct purposes:

• httpsOnly redirects HTTP requests to HTTPS.
• siteConfig.minTlsVersion defines the minimum TLS version accepted for inbound connections.
• siteScopedCertificatesEnabled enables per-app certificate resources.
• endToEndEncryptionEnabled encrypts front-end-to-worker traffic.

You can also create certificate child resources with Microsoft.Web/sites/certificates@2024-11-01, including importing a certificate by specifying its Key Vault resource ID and secret name. Use secure deployment parameters for any certificate password or PFX content; never store private key material or passwords in source control.

Azure CLI doesn't currently provide dedicated commands to manage Flex Consumption site-scoped certificates. Use the Azure portal or ARM/Bicep for certificate operations.

For more on using Bicep and ARM to configure site-scoped certificates in Flex Consumption see Automate resource deployment for your function app in Azure Functions.

Add and use a certificate

After creating the app with site-scoped certificates enabled, choose the workflow that matches your scenario.

Secure a custom domain with a managed certificate

In the Azure portal, open the Flex Consumption function app. Under Settings, select Custom domains, and then select Add custom domain and configure the certificates. For customer-managed certificates, add or import the private certificate under Settings > Certificates, and then bind it to the custom domain. A domain showing as Secured confirms that a binding exists, but clients can still reject an expired, self-signed, name-mismatched, or incomplete certificate chain. Test the endpoint from representative client environments.

Make a certificate available to function code

Adding a certificate to the app does not automatically expose it to code. In Settings > Certificates, open the menu for the required certificate and select Make accessible to app code.

This per-certificate control replaces WEBSITE_LOAD_CERTIFICATES in Flex Consumption. That app setting remains valid for other supported Functions and App Service hosting plans, but it isn't used by Flex Consumption. The per-certificate model applies least privilege more naturally: expose only the certificates that the workload must read.

Flex Consumption runs on Linux, so accessible certificates are files rather than entries in a Windows certificate store:

Load the file by thumbprint using the certificate APIs for your language. Don't assume a Windows certificate store is available, and don't package private keys with the function code.

For Key Vault imports, the platform places a renewed certificate version on the function instances after synchronization. If your application hard-codes a thumbprint, account for the fact that renewal produces a new thumbprint. Prefer a configuration or discovery pattern that lets you rotate without a code release.

Enable mutual TLS without confusing transport and identity

For inbound mTLS, configure the function app to request a client certificate by using the appropriate client-certificate mode: Required, Optional, or Optional Interactive User.

Azure terminates the client TLS connection at the platform front end and forwards the presented certificate to the application in the X-ARR-ClientCert header. The platform forwards the certificate; your function code remains responsible for validating it. Validation should include the trusted issuer and chain, validity period, intended usage, revocation policy where applicable, and the application's authorization rules. Do not trust the presence of the header alone.

If you use exclusion paths or Optional Interactive User mode, review the TLS renegotiation behavior carefully. TLS 1.3 and HTTP/2 don't support renegotiation, and requests with bodies larger than 100 KB can fail in configurations that require renegotiation. Prefer a mode and route design that avoids renegotiation when possible.

For outbound mTLS, make the private certificate accessible to app code, load it from /var/ssl/private, and attach it to the TLS client used to call the downstream service. This is separate from enabling inbound client certificates.

Operational guidance for production

Treat certificates as rotating dependencies rather than static configuration:

• Inventory certificate purpose, owner, source, expiration, bound hostnames, and whether code access is enabled.
• Use managed identity and Key Vault for centrally governed certificates.
• Alert before expiration for directly uploaded certificates and rehearse renewal.
• Monitor custom-domain DNS because managed certificate issuance and renewal depend on successful domain validation.
• Avoid certificate or issuer pinning when using App Service Managed Certificates.
• Test certificate rotation on all scaled instances, not only one warm worker.
• Keep HTTPS-only enabled and require TLS 1.2 or later unless a documented compatibility requirement prevents it.
• Use Azure Policy and deployment validation to keep transport-security settings consistent across environments.

When migrating from the legacy Consumption plan, remember that WEBSITE_LOAD_CERTIFICATES isn't used in Flex Consumption. Re-add the certificates to the new app, explicitly grant code access where required, and update code that expects the Windows certificate store to load files from the Linux paths instead.

Get started

The general availability of site-scoped certificates and end-to-end TLS gives teams a stronger security foundation for serverless APIs and event-driven applications on Flex Consumption. You can now keep certificate ownership at the function-app boundary, automate certificate lifecycle through Azure Key Vault or App Service, and encrypt request traffic through the worker that executes your code, all while retaining dynamic scale and consumption-based billing.

Use the following resources to begin:

• Azure update: TLS/SSL certificate and end-to-end TLS encryption support for Azure Functions Flex Consumption
• Create and manage function apps in a Flex Consumption plan
• Azure Functions scale and hosting options: certificate support
• Automate Azure Functions resource deployment
• TLS in Azure App Service
• Configure TLS mutual authentication
• Flex Consumption site-scoped certificates sample