Evaluating AI operations agents against your real support case data
Score your own support cases by addressability to build an evidence-based AI ops agent roadmap
Evaluating AI operations agents against your real support case data
By Kiran Khambete and Krish Balaraman · AWS Enterprise Support
Content level: Intermediate · ~14 minute read · Feature snapshot: June 2026
In the Turning AWS Support Cases into Operational Intelligence with the Coding Agents You Already Have ) The SRE team had answered the first question — where is the time going? — with a pipeline that classified closed support cases and measured the effort each consumed. The ranked heatmap lived on a shared dashboard. And then nothing happened. A dashboard, no matter how precise, does not reduce the next on-call rotation. This is the story of what happened next: how the same team took that ranked list of operational pain and turned it into an evidence-based adoption roadmap for an AI operations agent — using their own case data as the evaluation dataset, not a vendor demo.
⚠️ A note on numbers in this article. All percentages, hour estimates, and effort figures are representative — illustrative of the patterns one team observed in their specific case history. They are not benchmarks, guarantees, or universal ratios. Your results will reflect your workload mix, tooling maturity, and case volume. Run the pipeline on your own data to get numbers that mean something for your environment.
The ranked list that doesn't fix itself
The VP of Engineering had what they asked for. The SRE team's case-intelligence pipeline (covered in the companion article, Turning AWS Support Cases into Operational Intelligence with the Coding Agents You Already Have) had produced a ranked table: container orchestration failures at the top, deployment troubleshooting close behind, then networking, IAM confusion, data-pipeline failures, capacity errors — each with a measured effort figure in engineering-hours per quarter.
The VP's next question was immediate and uncomfortable: "Great — now what do we actually do about it?"
The team looked at the list. "Hire more" was not an answer — you cannot out-hire recurring operational toil, and scaling engineers linearly against recurring issues means you lose the scaling game before you start. "Build runbooks" helped for the most predictable patterns, but runbooks don't investigate — they execute after a human has already found root cause.
Someone on the team had been watching a different thread: the rise of AI operations agents — systems that don't wait for a human to correlate CloudWatch metrics with deployment and configuration changes recorded in CloudTrail and EKS pod events, but do it autonomously when an alert fires. The question wasn't "is this interesting?" — it was "how much of our actual, measured toil could one of these agents realistically handle, and how much remains unvalidated assumption?"
The answer, the team realized, was already in the data.
A new question: "Could an agent have handled this?"
A senior SRE pulled up a closed case from the top of the heatmap. An EKS worker node had gone NotReady. The case correspondence showed the investigation path: an engineer correlated CloudWatch CPU and memory metrics, pulled kubelet logs, found a recent ConfigMap change deployed the previous evening that introduced a resource limit that starved the node's system pods, identified the offending commit, rolled it back, confirmed recovery. Four messages from the customer, six from the engineer, twelve internal messages among three team members. The measured active resolution time: several hours of wall-clock engineering effort for what was — in hindsight — pattern matching across four data sources.
She wrote a note to the team: "This is correlation across metrics, logs, events, and deployment history. If an agent had access to the same data sources, it could have reached the same diagnosis. Maybe in minutes."
Then she pulled a second case of a different shape. A TLS certificate on an internal load balancer had expired over a weekend, taking a service down until someone was paged at 2 a.m. There was nothing to investigate here — the failure was entirely preventable. Had a scheduled check been running — the kind of periodic health routine an agent can execute on a fixed cadence, the way a team runs a standard operating procedure (SOP), scanning certificate expiry across the fleet and flagging anything inside a 30-day window — the case would never have opened. The first case showed where an agent could investigate faster; this one showed where an agent running a recurring check could stop the page from ever firing.
Those observations became the team's next project: go back through the classified case backlog and add one new column — addressability. For each case category, score whether an AI operations agent could have handled it fully, partially, or not at all. Then weight those scores by the effort hours already measured. The result would answer the VP's question with a number, not a hope.
The end-to-end process at a glance
Before diving into the rubric and the code, here is the whole pipeline — from raw support cases to a measured before/after outcome. The first two stages come from the companion article; this article is everything from stage 3 onward.
COMPANION ARTICLE (Part 1) THIS ARTICLE (Part 2)
┌───────────────────────────────┐ ┌─────────────────────────────────────────────────┐
┌──────────────┐ ┌────────────┐ ┌──────────────┐ ┌──────────────┐ ┌────────────┐
│ 1. FETCH │ │ 2. CLASSIFY│ │ 3. SCORE │ │ 4. WEIGHT │ │ 5. MATRIX │
│ closed cases │──▶│ + MEASURE │──▶│ addressabil- │──▶│ by measured │──▶│ FULLY / │
│ (Support API │ │ effort │ │ ity per │ │ effort hours │ │ PARTIALLY /│
│ / MCP) │ │ (ART) per │ │ category │ │ per category │ │ NOT │
│ │ │ category │ │ (LLM+rubric) │ │ │ │ │
└──────────────┘ └────────────┘ └──────┬───────┘ └──────────────┘ └─────┬──────┘
│ │
┌─────────────────┘ │
▼ ▼
┌──────────────┐ ┌─────────────────────────┐
│ Deterministic│ │ 6. RANKED ACTION LIST │
│ validation │ (rejects bad JSON, │ (a hypothesis, ranked │
│ + manual │ routes LOW-confidence │ by measured effort) │
│ review queue │ to human review) └────────────┬────────────┘
└──────────────┘ │
▼
┌─────────────────────────────┐
│ 7. DEPLOY on top category, │
│ MEASURE before/after MTTR │
│ → hypothesis proven or not │
└─────────────────────────────┘
└───────────────────────────────────────────────────────────────────────────────────────┘
Stages 1 and 2 come from the companion article. Each remaining stage maps to a section below — the story arc explains why, and "Building this yourself" gives the code:
| Stage | Read it in |
|---|---|
| 3. Score addressability (LLM + rubric) | Defining what "addressable" means (the rubric) → Scoring the case backlog (the logic) → Building this yourself: The scoring script (the code) |
| Deterministic validation + manual-review branch | Scoring the case backlog (validate_addressability) |
| 4. Weight by measured effort | Building this yourself: Aggregation — the effort-weighted matrix |
| 5. Addressability matrix | Reading the scorecard honestly (interpretation) → Building this yourself: Aggregation (the code) |
| 6. Ranked action list | Building this yourself: Output — a directional action list |
| 7. Deploy + measure before/after | Week one to quarter one: acting on the data |
Stages 3 to 5 are LLM-assisted but checked by deterministic validation. Stages 6 and 7 are where you test the hypothesis for real — deploy the agent and measure the before and after. The rest of this article walks each stage in order.
Defining what "addressable" means — before picking a product
The team's first instinct was to sign up for a demo and see what the vendor showed. The senior SRE pushed back: "If we let the vendor define what's addressable, we'll get an answer optimized for their slide deck, not our workload."
Instead, the team defined their own evaluation rubric — a fixed set of scoring criteria applied identically to every case category, so the judgments stay consistent and comparable rather than ad-hoc. Theirs was vendor-neutral, anchored in what an AI operations agent should be able to do based on the team's own operating model.
They wrote it on a whiteboard as five capability levels:
Level 1 — Investigate autonomously. Given an alert or symptom, correlate signals across metrics, logs, traces, and deployment history to identify probable root cause — without a human directing the search.
Level 2 — Detect proactively. Run scheduled checks that catch configuration drift, resource exhaustion trends, certificate expiry, and parameter misconfigurations before they page someone.
Level 3 — Propose remediation. Generate a specific mitigation plan — a set of steps, a config change, a rollback command — and present it to a human for approval before execution.
Level 4 — Execute known-good fixes. For well-understood, previously validated patterns, execute the remediation autonomously — update a parameter, scale a resource, restart a service — without waiting for human approval.
Level 5 — Know its limits. Recognize when the investigation has reached a boundary the agent cannot cross — a novel failure mode, an external dependency, a decision that requires business context — and escalate cleanly to a human with the investigation context already assembled.
The team recognized: an agent that only hits Level 1 is still valuable — it compresses investigation time. An agent that hits Levels 1–3 but never Level 4 still returns hours per quarter. Partial addressability is not a disqualifier; it is the realistic operating point for most cases.
Scoring the case backlog: where would the agent have helped?
The team took their existing classified output — each case with a category, an automatable flag, a recurring signature, and a measured effort in minutes (one way to derive that figure: compute Active Resolution Time from case correspondence timestamps — for chat cases, the span from first to last message; for email cases, group messages into work sessions on a 60-minute idle gap and sum them, with per-severity caps so a case left open over a weekend isn't counted as continuous effort) — and added the addressability column. The pipeline extension was small: one new LLM call per case category (not per case — there were only a dozen categories), scored against the agent's documented capabilities.
The critical design decision: score against public documentation only. No roadmap promises. No demo claims. No "our sales engineer said it could do this." The LLM reads the agent's user guide, the case category's definition, and the representative case examples in that category, then emits a judgment.
This step has two halves that work together. The prompt below is the LLM input — it carries the rubric the team whiteboarded and pins the exact JSON shape the model must return. The Python that follows it (validate_addressability) is the deterministic guardrail — it does not produce the judgment; it inspects whatever JSON the model emits and rejects anything that doesn't conform. Read the prompt as "what we ask the model to decide," and the code as "what we refuse to accept from the model's answer."
First, the prompt (all code blocks in this article are Python):
ADDRESSABILITY_RUBRIC = """ You are evaluating whether an AI operations agent could have handled cases in this category. You are given: 1. The category name and description 2. Representative case summaries from this category 3. The agent's DOCUMENTED capabilities (from its public user guide) Score the category against each addressability level: - FULLY: The agent could handle this end-to-end with no human intervention beyond initial configuration. - PARTIALLY: The agent could significantly accelerate investigation or propose remediation, but a human must validate or complete the resolution. - NOT_ADDRESSABLE: The agent lacks the data access, judgment, or authority to materially help with this category. Return ONLY JSON: { "addressability": "FULLY | PARTIALLY | NOT_ADDRESSABLE", "level_reached": 1-5, "reasoning": "<2 sentences: what the agent can and cannot do for this category>", "confidence": "HIGH | MEDIUM | LOW" } RULES: - Score ONLY against documented capabilities. If you are unsure whether the agent supports something, score it NOT_ADDRESSABLE. - A case requiring external vendor coordination, billing negotiation, or architectural decision-making is NOT_ADDRESSABLE regardless of the agent's investigation ability. - If the agent can investigate but cannot apply the fix (e.g., fix requires a service-team backend change), score PARTIALLY — investigation acceleration still has measurable value. """
The rubric is parameterized: swap the agent capability section for any AI operations agent you are evaluating. The pipeline structure stays the same.
Deterministic validation is the guardrail half — it follows the LLM call, applying the same principle from the classification step. The code rejects any addressability value not in the allowed enum, any level_reached outside 1–5, and — critically — any claim the team cannot trace back to a specific documentation page. A LOW confidence score routes the category to manual review rather than accepting the LLM's judgment unchecked.
VALID_SCORES = {"FULLY", "PARTIALLY", "NOT_ADDRESSABLE"} def validate_addressability(result: dict, category: str) -> dict | None: if not isinstance(result, dict): return None if result.get("addressability") not in VALID_SCORES: return None level = result.get("level_reached") if not isinstance(level, int) or not 1 <= level <= 5: return None if result.get("confidence") == "LOW": result["_needs_review"] = True # route to manual queue result["category"] = category return result
The validated scores then feed an effort-weighted matrix — one row per case category, with its addressability score and the effort hours it historically consumed. To keep the code in one place, the join with measured effort and the aggregation logic are shown once, as production code, in the Building this yourself section below.
These scores reflect this team's case mix — container-heavy, multi-account, financial services. An organization running mostly serverless, or one with a different observability stack, would see a different distribution.
Putting a real agent to the test: AWS DevOps Agent
The rubric was vendor-neutral; the evaluation needed a concrete candidate. The team chose AWS DevOps Agent — production operations features generally available since March 2026, release management in preview — with their organization already on AWS Enterprise Support. That last point mattered for cost: DevOps Agent is included with paid AWS Support plans through monthly credits (a percentage of the support charge, scaled by plan tier), so the team could start evaluating against their own cases at little incremental cost. The question was: how does this specific agent score against our addressability rubric, on our actual case categories?
They fed the agent's documented capabilities into the scoring prompt and ran the evaluation. The results mapped to their top case categories like this:
The node failure case, replayed
The EKS NotReady case that had taken multiple engineers several hours became the team's first test. DevOps Agent's autonomous incident response works exactly the way the senior SRE had hypothesized: triggered by a CloudWatch alarm or a PagerDuty alert, the agent federates across metrics, kubelet logs, deployment history, and Kubernetes events — the same four data sources the engineers had correlated manually. It produces a root cause analysis and a mitigation plan. The investigation that took hours of wall-clock time completes in minutes.
Score: PARTIALLY (Level 3). The agent investigates and proposes the rollback. A human validates before the rollback executes — which is exactly what the team wants for production config changes in a regulated environment.
Custom agents for recurring patterns
Several of the team's case categories were not incidents at all — they were recurring operational tasks that became cases only because nobody caught the drift early enough. Certificate expiry approaching without renewal. Database parameters drifting from baseline. Slow queries accumulating without review.
DevOps Agent's custom agents — a June 2026 addition — let the team create scheduled agents within their Agent Spaces. The team used these for SRE-style health checks: a daily agent checks certificate expiry across their fleet. Another reviews slow-query logs and flags parameters that deviate from the team's baseline. A third scans deployment configs against the team's standards. (See the announcement for additional use cases.)
These are not investigations — they are proactive detections that prevent the case from being opened in the first place.
Score: FULLY (Level 2 → preventing the incident). No human in the loop for the check; humans are notified only when something needs attention.
Extending reach via Model Context Protocol (MCP)
The team uses Datadog for application-level observability and PagerDuty for alerting. Without access to this telemetry, the agent would be limited to AWS-native signals — still useful, but blind to the application layer where many of their cases originate.
DevOps Agent's remote server connectivity — supporting both MCP and direct integrations for Datadog, Dynatrace, Grafana, New Relic, Splunk, and PagerDuty — connects the agent to the same observability context the engineers use. When an investigation triggers, the agent can pull application metrics, traces, and logs from Datadog alongside AWS infrastructure signals.
This extended the "PARTIALLY" cases further toward "FULLY" — the agent's investigation is now grounded in the same data the engineers would use, not a subset.
Agent-to-Agent (A2A) delegation for internal tools
One category stubbornly stayed at NOT_ADDRESSABLE in the first pass: cases that required correlation with the team's internal configuration management database (CMDB) — a service DevOps Agent has no built-in connector for.
The team used the A2A protocol for remote agents to build a lightweight remote agent that wraps their CMDB's API. When DevOps Agent's investigation needs CMDB context — which service owns this resource, what was the last change window, who is the responsible team — it delegates the lookup to the remote agent and incorporates the response into its investigation.
The team built the remote agent in a day. Three case categories moved from NOT_ADDRESSABLE to PARTIALLY.
Release management (preview)
The team noticed a recurring signature in their deployment-related cases: a code change introduced a configuration that violated one of their operational standards, wasn't caught in review, deployed to production, and became an incident. The investigation was straightforward once someone looked — but the looking took hours, and the incident had already impacted customers.
DevOps Agent's release management capability (preview, June 2026; see announcement) assesses code changes against natural-language standards the team defines, and runs autonomous release testing in production-like environments. The deployment-category cases that traced back to "standards violation missed in review" would, with this capability enabled, be caught before production.
Score: FULLY (preventing the case from existing). Currently in preview — the team noted this dependency in their roadmap.
The capabilities listed here reflect generally available and preview features as of June 2026. Verify current availability, regional coverage, and pricing against the AWS DevOps Agent documentation and the what's new page before adopting.
Reading the scorecard honestly
The team assembled the addressability matrix, weighted by measured effort, and brought it to the VP. The numbers told a nuanced story — representative of their specific workload (container-heavy, multi-account, financial services):
-
Fully addressable: roughly one-third of their quarterly case-effort hours. These are cases the agent could handle end-to-end — either by autonomous investigation + approved remediation, or by proactive detection that prevents the case entirely.
-
Partially addressable: roughly two-fifths. The agent accelerates the investigation significantly — correlating signals, identifying probable root cause, proposing a mitigation plan — but a human validates the diagnosis or executes a remediation that requires business judgment.
-
Not addressable: the remainder. Billing negotiations. Architectural design discussions. Novel failure modes the agent has never seen. Cases requiring coordination with external vendors or AWS service teams for backend fixes. Cases where the diagnosis depends on data no agent can reach.
The senior SRE framed it for the VP: "We're not replacing anyone. We're giving the on-call engineer a first responder that's already done the correlation work before they open their laptop. And for the proactive categories — we're preventing the page from happening."
The VP asked the right follow-up: "What's the confidence on these numbers?"
The team was honest: "The addressability scores are our judgment, validated against documentation — not a proof of performance. The effort hours behind them are measured from timestamps, not estimated. The real test is deploying for the top category and measuring the before/after. These numbers are the hypothesis; the next quarter is the experiment."
Your distribution will differ — possibly significantly. An organization with mostly serverless workloads, a different observability stack, or a heavier proportion of architectural guidance cases will see a different profile. The value is in running this against your own data, not in treating these representative ratios as a benchmark.
Building this yourself
The evaluation pipeline the team ran is small. Here is what you need, assuming you already have classified case output (from the companion article or any structured export with category + effort fields):
Input: your classified case data
A JSON or CSV with at minimum:
[ { "case_id": "case-EXAMPLE-0001", "category": "container-orchestration", "recurring_signature": "EKS node NotReady after config change", "automatable": true, "art_minutes": 247.0 } ]
The capability document
Assemble the agent's documented capabilities into a structured reference the scoring prompt can consume. This string is not standalone code — it is an input that gets injected into the scoring prompt. In the score_categories function further down, it fills the ## Agent Capabilities section of every prompt sent to the model, sitting alongside the rubric and the category being evaluated. Swapping this one block (and nothing else) is what lets the same pipeline evaluate a different agent. For AWS DevOps Agent, it maps directly to the user guide:
AGENT_CAPABILITIES = """ DOCUMENTED CAPABILITIES (source: AWS DevOps Agent User Guide, June 2026): 1. AUTONOMOUS INCIDENT INVESTIGATION # maps to: "The node failure case, replayed" — scored PARTIALLY (Level 3) - Triggered by: CloudWatch alarms, PagerDuty, ServiceNow, webhooks, observability-tool events - Data sources: CloudWatch (metrics, logs, Logs Insights, Container Insights), CloudTrail, AWS resource configurations, EKS control plane, ECS (task/service events), S3 (logs/artifacts) - Third-party observability: Datadog, Dynatrace, Grafana, New Relic, Splunk, Prometheus (via direct integration or custom MCP servers) - Output: root cause analysis, mitigation plan, agent-ready instructions 2. APPLICATION TOPOLOGY MAPPING # maps to: cross-service investigations — improves root-cause quality in # the node-failure and MCP/A2A scenarios (supporting capability) - Auto-discovers resources and relationships - Correlates topology with telemetry during investigations - Enables cross-service root cause tracing 3. PROACTIVE INCIDENT PREVENTION # maps to: the periodic certificate-expiry / drift example — supports the # FULLY (Level 2) "prevent the case from opening" scenarios - Analyzes historical investigation patterns - Recommends improvements: observability, infrastructure, deployment pipeline, application resilience - Recommendations include implementable specifications 4. CUSTOM AGENTS (June 2026) # maps to: "Custom agents for recurring patterns" — scored FULLY (Level 2) - User-defined scheduled agents within Agent Spaces - Run on cadence (daily, hourly, custom) - Use cases: database health reports, log anomaly flagging, config drift 5. RELEASE MANAGEMENT (Preview, June 2026) # maps to: "Release management (preview)" — scored FULLY, but gated on # preview availability; re-confirm GA status before relying on it - Assesses code changes against natural-language standards - Runs autonomous release testing in production-like environments 6. TOOL INTEGRATION # maps to: "Extending reach via MCP" and "A2A delegation for internal tools" # — moves PARTIALLY cases toward FULLY and rescues NOT_ADDRESSABLE ones - Model Context Protocol (MCP) servers (custom), A2A remote agents - Code repos: GitHub, GitLab, Azure DevOps - Ticketing: ServiceNow, PagerDuty - Outbound: EventBridge lifecycle events, Slack summaries DOCUMENTED LIMITATIONS: - Cannot access external vendor systems or carrier-side data - Cannot make billing or account-level changes - Release management is in preview — not GA """
Treat this capability document as a living artifact, not a one-time write-up. The June 2026 feature wave (custom agents, release management, MCP/A2A connectivity) is unlikely to be the last — re-derive the block from the user guide and the what's new page on each re-scoring pass, adding new capabilities and retiring preview caveats as features reach GA. The scores are only as current as the capabilities you feed the prompt.
The scoring script
import json from pathlib import Path def score_categories( categories: list[dict], # unique categories with representative case summaries agent_capabilities: str, # the capability document above rubric: str, # ADDRESSABILITY_RUBRIC from earlier classify_fn # your LLM call function (Bedrock Converse, coding agent, etc.) ) -> list[dict]: """Score each case category against the agent's documented capabilities. Note: for production use, wrap classify_fn with retry logic and a timeout. """ scored = [] for cat in categories: prompt = ( f"{rubric}\n\n" f"## Agent Capabilities\n{agent_capabilities}\n\n" f"## Category to Evaluate\n" f"Name: {cat['name']}\n" f"Description: {cat['description']}\n" f"Representative cases:\n{json.dumps(cat['examples'], indent=2)}\n" ) result = classify_fn(prompt) # returns parsed JSON validated = validate_addressability(result, cat['name']) if validated: scored.append(validated) else: scored.append({ "category": cat['name'], "addressability": "NEEDS_REVIEW", "confidence": "LOW", "_needs_review": True }) return scored
Aggregation: the effort-weighted matrix
import pandas as pd def aggregate_addressability(scored: list[dict], case_data: pd.DataFrame) -> pd.DataFrame: """Join addressability scores with measured effort and produce the matrix.""" # Sum effort hours by category from your case data effort = (case_data.groupby("category")["art_minutes"] .sum().div(60).round(1) .rename("quarterly_hours")) scores_df = pd.DataFrame(scored) matrix = scores_df.merge(effort, left_on="category", right_index=True, how="left") matrix["quarterly_hours"] = matrix["quarterly_hours"].fillna(0) # Summary by addressability level summary = (matrix.groupby("addressability")["quarterly_hours"] .agg(["sum", "count"]) .rename(columns={"sum": "total_hours", "count": "categories"})) return matrix.sort_values("quarterly_hours", ascending=False), summary
Output: a directional action list
aggregate_addressability returns a (matrix, summary) tuple, so unpack it before passing the matrix on — matrix, summary = aggregate_addressability(scored, case_data). The action list itself is not a business case — it is a hypothesis ranked by measured effort:
def action_list(matrix: pd.DataFrame) -> list[dict]: """Produce a ranked list of actions from the addressability matrix.""" actions = [] for _, row in matrix.iterrows(): if row["addressability"] == "FULLY": actions.append({ "priority": "HIGH", "action": f"Deploy: configure agent for '{row['category']}'", "potential_hours": row["quarterly_hours"], "note": "Upper-bound opportunity — measure actual reduction after deployment" }) elif row["addressability"] == "PARTIALLY": actions.append({ "priority": "MEDIUM", "action": f"Extend: connect MCP/A2A tools for '{row['category']}'", "potential_hours": row["quarterly_hours"], "note": "Agent accelerates investigation; human validates. " "Actual time savings will be a fraction of total hours." }) # NOT_ADDRESSABLE categories omitted — no agent action to take return sorted(actions, key=lambda x: x["potential_hours"], reverse=True)
Treat the potential_hours as a ceiling, not a commitment. Fully addressable means the agent could handle it — not that time savings equal 100% of historical effort. Partially addressable means acceleration — the savings are a fraction, discoverable only by measuring the before/after. And note the matrix ranks by effort recovered, not net ROI: it tells you where to look first, not what you net after agent cost. Weigh the opportunity hours against that cost — for AWS DevOps Agent, Support-plan credits offset much of it; a commercial AIOps platform carries license cost.
Week one to quarter one: acting on the data
The team did not attempt to deploy everything at once. The data told them where to start:
Week 1 — They picked the highest-effort fully addressable category (recurring EKS health checks that generated cases only because drift was caught late) and deployed a custom agent: a daily Kubernetes health scan across their clusters that flags NotReady nodes, pod restart loops, and resource limits approaching saturation. Cases in this category dropped to near-zero within days — not because incidents stopped, but because the agent caught them before they became incidents.
Month 1 — They connected their Datadog MCP server and PagerDuty integration, extending the agent's investigation reach into their top partially addressable categories. On-call engineers started receiving investigation reports before they opened their laptops — the correlation work pre-done, a mitigation plan proposed, "approve Y/N?" The time from page to resolution compressed.
Quarter 1 — The team measured actual mean time to resolution (MTTR) for the categories where the agent was deployed, compared against the same categories' baseline from the quarter before deployment. The addressability matrix had given them a hypothesis. The before/after comparison proved it for their environment — and disproved it for one category where observability gaps meant the agent's investigations were incomplete. That gap became a backlog item: improve logging for that service, then re-enable the agent.
The VP got the answer in their own unit — engineering-days recovered per quarter, with a clear trace from the case data through the addressability score to the measured outcome.
Swapping the agent: this pattern is not vendor-locked
The pipeline the team built has one parameterized input: the agent capability document. Everything else — the rubric, the scoring prompt, the validation logic, the aggregation, the effort measurement — is agent-agnostic.
To evaluate a different AI operations agent:
- Replace
AGENT_CAPABILITIESwith that agent's documented capabilities (from its user guide or API reference — not from marketing materials). - Re-run the scoring step. Same rubric, same categories, same effort data.
- Compare the resulting addressability matrices side by side.
The team used this property to evaluate a second candidate — a commercial AIOps platform that one of their peer teams had adopted. The scoring prompt was identical; only the capability document changed. The results highlighted where each agent was stronger and where both had gaps, which gave the team a clear basis for a "build-vs-buy" conversation grounded in their own workload, not in a vendor's reference architecture.
What this pattern doesn't solve
Agent capabilities evolve. The June 2026 feature wave added custom agents, release management, MCP/A2A protocols, and five new Regions to AWS DevOps Agent — capabilities that didn't exist when this team started their evaluation. Re-score periodically. An annual evaluation on a service that ships monthly will be stale by the time you act on it.
Partial ≠ guaranteed. An agent scored "PARTIALLY" for a category can only accelerate investigation if the observability data exists. If your application doesn't emit traces, or your logs are unstructured, the agent's investigation will be shallow. Addressability is a ceiling; your instrumentation determines how close you get.
The model's judgment is an estimate. The addressability scoring step uses an LLM — the same honest asymmetry from the companion article. The same category might score FULLY on one run and PARTIALLY on another if the representative cases are borderline. Pin a model version, validate against documentation, and treat the scores as informed estimates subject to human review — especially for the NOT_ADDRESSABLE boundary.
Novel failures are not scoreable. If a failure mode has never appeared in your case history, it won't be in your classified data, and this pattern won't evaluate it. The pattern scores what you have experienced. For what you haven't, architectural resilience reviews and chaos engineering remain the tool.
This does not replace a proof of concept. The addressability matrix gives you a data-grounded hypothesis about where to start. Proving the hypothesis requires deploying the agent on your top category and measuring the real before/after. The matrix reduces the risk of starting in the wrong place — it does not eliminate the work of validating in production.
Key takeaway
The team that started with "where is the time going?" now has "here is how much we can get back, and here is the evidence trail." The case backlog that was a liability — write-once records accumulating in an API — became the evaluation dataset. The adoption roadmap wrote itself from the data: start where the effort is highest and the addressability is clearest, extend where the agent needs more context, and leave alone what requires human judgment.
No vendor chose the starting point. The team's own operational history did — scored against documented capabilities, weighted by measured effort, validated by a before/after comparison in their own environment.
The pipeline is small, the principle is simple, and the data is already yours. Fetch it, classify it, score it, and let the numbers tell you where to start.
References
- AWS DevOps Agent — About, Autonomous incident response, Proactive incident prevention, Custom agents, Release management (preview), Remote servers (MCP/A2A), What's new
- What's new announcements — Custom agents (June 2026), Release management (June 2026), General availability (March 2026)
- AWS DevOps Agent — Pricing and Using AWS DevOps Agent with your AWS Support plan
- AWS Support API — DescribeCases, DescribeCommunications
- AWS Support MCP server — Documentation, GitHub
- Model Context Protocol specification
- Agent-to-Agent (A2A) protocol on AWS
- Building an end-to-end agentic SRE using AWS DevOps Agent (AWS DevOps Blog)
- Production-ready autonomous incident resolution with AWS DevOps Agent and Datadog MCP Server (AWS DevOps Blog)
This article is shared with the community as a builder guide. All examples use representative data from a composite scenario — not a specific customer's environment. Verify service availability, regional coverage, feature GA status, and pricing against current AWS documentation before implementing. The addressability percentages and effort figures are illustrative of one team's workload mix and should not be interpreted as benchmarks or guarantees for your environment.
- Language
- English
Relevant content
- AWS OFFICIALUpdated 4 months ago
asked a year ago
asked 2 years ago
