Azure News - 2026-09-25

A common question that we get from our customers is "Hey guys, we are excited about SRE agent and we 've created one but whats the best way to configure it?". The below guide aims at sharing some of the best practices that anyone can follow to make the most out of our SRE Agent. This guide is based on SRE Agent's own scoring system and we evaluated production telemetry of the agents performing best vs. the ones performing the worst and we tried understanding the key differences that led to one agent performing better than the other.

Interestingly, agents doing the best work are usually not the most heavier ones. Some of the strongest agents we looked at had just one skill. On the other hand, some of the worst performing agents had large skill libraries, plenty of connectors, and insane number of tools. The difference almost always comes down to one thing: "whether the right work reaches the right handler"

This guide walks through what to set up, roughly in the order you'd do it. It's written for someone configuring an agent for the first time, so it starts simple.

The seven things that matter

These are written in the order you'll most likely execute on them. "How much it matters" shows how much each aspect makes an agent significantly better (the goal is to help you quickly decide which ones to prioritize).

Start here: The One idea that is worth internalising

Think of your agent less like an AI system but more like a new hire in your team, even though its hard to visulaize it that way. Do you burden a new hire with year old documents, PIR investigation reports, architecture documents that have drifted over years, information about systems that have become obsolete or telemetry logging that is long gone? The same hold true for an agent.

An agent needs access to your systems. They need to know which problems they should be really solving. They need true context about how your service works. And they need to know when to act and when to check with someone first.

If you get these things right, a fairly plain agent can do excellent work. Otherwise, you are just in a constant loop of updating prompts, changing tools, updating skills and getting frustrated. Everything below is a version of those four things.

1. Connect your data first

You might have heard this phrase before - "Context is the King". Before writing a skill or an agent prompt, "empower" your agent first: your telemetry logs (most important), your incident system, your work tracking, your repositories. Agent without the right data is never going to succeed no matter how good prompt or skill is. Connect the "right" connectors and not "many" connectors.

These are important so "do" these

• Connect telemetry and work tracking first — These help with most of the real investigative work.
• Add deployment history and service health next — this is what links a problem to the change that caused it.
• Index the systems where work already happens — your wiki and work items get used heavily; a separate document store barely gets touched.
• Connect Teams or email for delivery — findings should land where your team already works.
• Check connections are healthy, not just configured — a dead connector fails silently and takes your scheduled runs with it.

Avoid doing these

• Don't connect a system until you need it — extra connections rarely get used, and you still have to keep them working.
• Don't treat Teams or email as investigation tools — they carry a fraction of the traffic because they're output, not input.
• Don't treat past incidents as the answer — use them as a starting point, then confirm with current data.
• Don't add connections to fix a struggling agent — the best and worst agents we studied had nearly identical, fully healthy connection sets. One scored 4.9, the other 2.3. The difference was routing.

Never let the agent decide the source of truth. Feed it!

2. Start with the problem that annoys you most

This goes back to the same new hire principal. Don't try to teach the agent your whole system. Pick an important component that wakes people up, or the investigation everyone dreads because it takes four hours and always ends the same way.

Work that one scenario with the agent, interactively. The best to do is to "chat" with the agent without writing a new custom agent. Let it dig, correct it when it goes the wrong way, point it at the data it didn't knew about. When you and and the agent solve the issue together, ask the agent to turn what just happened into a skill or a custom agent. This approach is way more effective that perfecting the "correct" agent by starting with a lengthy prompt.

Once you perfected one problem, move on to the next one.

Each scenario is small, so it's quick to get right, and the context compounds. After a few weeks, you'll notice that the agent is helping with things you never "explicitly" taught it. It just learns (just like a good new hire).

3. Send the right incidents to the right specialist

If you read only one section, read this one. Sending an agent the wrong work was one of the clearest problems we found.

A response plan decides which incidents reach an agent. If the plan accepts every incident but sends them all to a specialist built for one problem, many incidents will arrive without the information that specialist needs. It stops without helping, and the incident is marked unresolved. The agent may look broken, but it was given work it wasn’t built to do.

We saw this with an agent that handled scheduled work well, scoring 4.5 out of 5, but scored 1.7 out of 5 on incidents it received through a broad response plan. Another agent used 71 response plans, each directing a specific kind of incident to the right specialist. It scored 4.5 out of 5 on incidents, with fewer than 1% failing. These are examples from different agents, not a before-and-after test, but they show why it’s worth checking how incidents are routed.

We recommend using response plans that match specific incidents to specialists that can handle them. Having many focused plans is fine; sending every incident through one broad plan to a narrow specialist is the problem.

4. Write a few skills that really teach something

The number of skills an agent has matters less than what those skills teach it.

In our study, some agents with no skills performed better than agents with several. The strongest results were associated with skills that gave clear, detailed guidance, while agents with vague or empty skill descriptions struggled. One agent had three skills with no descriptions at all. That doesn’t prove that writing more will improve results, but it does show why adding skills just to increase the count isn’t useful.

Think of a skill as a briefing for a colleague - It should explain which question it answers, the words people might use when asking it, how to decide what to do, and when the skill does not apply. It should also warn about mistakes that are easy to make. We recommend writing a skill when you have specific knowledge or judgment to teach the agent. Make it detailed enough to guide a real decision, rather than adding a short description that only names a topic.

Here are three examples of how to turn a vague skill description into guidance the agent can actually use. These are illustrative examples, not descriptions taken from the study.

Billing issue

• Too vague: “Helps with billing problems.”
• Useful: “Use this when someone asks whether an order was charged, whether a missing charge will appear later, or how to recover a failed charge. Check the order and billing records before deciding. Don’t suggest creating a new billing record until you’ve ruled out an existing or delayed charge.”

Deployment failure

• Too vague: “Investigates failed deployments.”
• Useful: “Use this when a deployment has failed or stopped making progress. Check its status, the failed step, and recent changes. Don’t recommend retrying until you know whether the previous attempt is still running or left work unfinished.”

High error rate

• Too vague: “Handles service alerts.”
• Useful: “Use this when a service’s error rate rises. Check when the increase started, which requests are failing, and whether a deployment or dependency changed around the same time. Don’t blame the most recent deployment without evidence that it affected the failing requests.”

The useful versions tell the agent when to use the skill, what to check, and which mistake to avoid.

5. Add custom agents when different problems need different help

A custom agent has its own instructions, tools, and scope for a particular kind of work. Create one when a problem comes up often enough to need a clear owner—not just because the option is available.

For example, a deployment custom agent could check rollout status, failed steps, and recent changes when an alert follows a release. A billing custom agent could check order and charge records, with instructions to avoid actions that might charge a customer twice. These jobs need different evidence and different safeguards. An occasional question doesn’t necessarily need its own custom agent.

In our study, agents with one or two custom agents performed much like agents with none; agents with three or more tended to do better. That’s an observation, not a target number. Add them as distinct, recurring problems emerge.

Be clear about when each custom agent runs. As the Zero Ops guidance explains, you can call one directly in chat or select it through a response plan or scheduled task. This lets you send a deployment alert to the deployment custom agent without sending every incident there.

Limit access to what the job needs. Start with read access for investigation, and grant permission to make changes only where needed, with appropriate approval. Give each custom agent an explicit list of tools: for a custom agent, an empty tool list means default tool access, not “no tools.” The agent’s identity and permissions still set the outer boundary; a custom agent’s instructions alone are not a security control.

6. Decide how much freedom to give, scenario by scenario

Review mode protects you by asking a person to approve actions. Use it where approval matters, but remember that work can sit unfinished while it waits.

We saw this in two agents handling busy incident queues. Both followed their instructions well and rarely failed. One ran in review mode: 96% of its incident work was waiting for approval, and its overall score was 3.0 out of 5. The other worked autonomously and scored 4.8 out of 5. These are different agents, so the scores don’t prove that changing the mode alone would close the gap. They do show why it’s worth checking how much work is waiting.

Choose the level of approval for each kind of work. For example:

• Investigating an alert: Let the agent read logs, check service health, and compare the alert with recent deployments without asking for approval at every step.
• Sharing findings: If your team is comfortable with it, let the agent post an investigation summary to the incident. If posts need review, let it prepare a draft instead.
• Restarting a service: Have the agent explain what it found and ask a person to approve the restart.
• Changing a database or network setting: Keep a person involved. These changes can affect more than the incident the agent is investigating.

The goal isn’t to make the whole agent autonomous or put everything in review mode. Let routine investigation move forward, and require approval where the consequences of a mistake are greater.

7. Check the agent can finish the job

An agent can do a good investigation and still fail at the last step.

We saw several examples where an agent did useful work but couldn't complete the final step:

• One agent found three incidents that needed action, but the incident-management write tools it needed were unavailable. It could explain what to do, but couldn't complete its scheduled task.
• Another investigated an incident and found the right team to take over, but the transfer failed with an authorization error. The incident stayed unresolved.
• A third gathered evidence for an incident, but couldn't save a lasting update because it lacked permission to write back to the incident system.

These can look like investigation failures when the actual blocker is access. That doesn't mean granting every agent write permission: decide which final steps it should perform, give it only the access those steps require, and make the human handoff clear for anything it shouldn't do itself.

Before sending an agent to a live incident queue, try one example from start to finish. Check not only that it can find the answer, but that it can deliver the result in the way your team expects.

Troubleshooting: When something isn't working, find out why

Different problems can look alike. Check what happened before changing the agent's instructions.

Two questions are especially useful: Did the agent use the tools it needed? And do failures look the same each time? No tool use during an investigation may point to a routing problem. Repeated identical errors may point to a broken connection or missing permission. If the mistakes vary, look more closely at the instructions, skills, and information the agent received. These are clues, not diagnoses on their own.

If you already have an agent that's struggling

Check the basics before rewriting its skills:

1. Check routing. Are unrelated incidents reaching a custom agent built for a narrower problem?
2. Check connections. Can the agent actually reach the systems it needs?
3. Check what is waiting. Is review mode holding up work that could safely continue?
4. Check the last step. Can the agent post its findings or complete an approved action?
5. Then review its guidance. If the agent has the right work, access, and permissions but still makes different mistakes, improve its instructions or skills.

How we know this

We studied production SRE Agent activity. We set aside test and demo agents, looked at agents doing real scheduled work, incident response, and chat, and compared their results with how they were configured.

The examples show patterns we observed, not before-and-after tests. An agent scoring differently on incidents and scheduled work is a useful clue, but it does not prove that routing was the only cause. Scores can also change when the evaluation changes, so use the numbers as context rather than promises of what a configuration change will achieve.

Read scores in context. Compare incidents with incidents and scheduled tasks with scheduled tasks: open-ended chat often scores lower than work with a clear goal. Greetings aren't investigations, so don't use their scores to judge incident performance. And if an agent is marked down for a wordy answer, check whether its findings were useful before treating it as a failure.

This is a starting point for teams to review and improve and we really hope this helps!