Are we ready to open the lines? A custom SRE agent for the contact center risks your pipeline can't catch

⚠️ A note on the scenario and numbers. The contact center, the team, and every figure here are a composite — representative of patterns seen across several financial-services organizations, not any one customer's environment, and not benchmarks. AWS service behaviors, metrics, and API fields are described from public documentation as of June 2026; verify current availability, regional coverage, and feature status against AWS docs before you build. Examples use synthetic data.

In the previous article, an SRE team scored their support history against AWS DevOps Agent's documented capabilities, deployed it on their highest-effort category, and measured the before/after. For most of their estate it worked: incidents that used to take hours came back in minutes.

This is about the failures it couldn't help with — not because the agent wasn't good enough, but because nothing ever triggered it. And it's about the one thing that finally did.

The morning it broke

It was a Monday, and the bank's marketing team had launched a card-offer campaign over the weekend. Call volume into the contact center doubled by 9am — entirely expected, entirely planned for.

By 9:15, one in five callers who tried to authenticate couldn't. The contact flow's identity step — a Lambda "data dip," a mid-call lookup the flow runs to fetch the caller's details from the bank's CRM — was timing out, dumping authenticated callers into an overflow queue. Agents who should have been taking verified customers sat watching a queue of frustrated, unverified ones.

The post-incident review found the real story, and it was uncomfortable. Nobody had deployed anything. For the previous two weeks, the CRM behind that data dip had been getting slowly slower — p99 latency creeping up, a few more timeouts each day, as a connection pool quietly saturated under growing load. Every single day it was "amber, but within range." No alarm fired, because no threshold was crossed. Then Monday's volume doubled, the slow path tipped past the flow's timeout, and a two-week-old creep became a customer-facing outage at the worst possible hour.

Here is what made the senior SRE put her coffee down: none of this could have been caught before it happened by the tools they trusted. It wasn't in a pull request — no code changed. It couldn't be reproduced in a lower environment — there's no production traffic and no real CRM load there. And no single dashboard had crossed a red line — the error rate was "amber," the call volume was "normal for a Monday," and the flow's fallback rate was a number nobody watched. Each signal, alone, looked fine. Only together did they spell out: this was going to break, and we could have seen it at 7am.

What the contact center promises

A bank's contact center makes one daily, customer-facing promise, on two legs:

Inbound — when we open the lines, any customer who calls reaches us, is identified, is served, and the call is recorded for compliance — at the volume the day brings.

Outbound — when we call a customer (a fraud check, an alert, collections), the call gets through and isn't blocked as spam.

That promise names no AWS service — a customer just wants to call their bank and get helped. It collapses to one question a contact center manager already asks every morning: "Are we ready to open the lines today?" The answer depends on several things, across several teams, that nobody checks together. Here's the path it runs on:

 INBOUND   customer call ─▶ Amazon Connect (flows, routing, recording)
                              ├─▶ Amazon Lex (self-service IVR)
                              └─▶ AWS Lambda (data dip) ─▶ CRM / database (identify the caller)
           recordings ─▶ Amazon S3 (compliance)

 OUTBOUND  Connect outbound ─▶ carrier network ─▶ customer
           (the bank's caller-ID numbers carry a carrier reputation)

Why no single tool catches it

Your pipeline catches problems inside a change — a bad config, a missing permission, a broken flow. The failures in this article are a different kind: they come from time, traffic, and the outside world, with no change at all —

• a downstream whose error rate slowly creeps under real load,
• a quota you drift toward as volume grows,
• an outbound number a carrier starts flagging as "Scam Likely,"
• recordings that quietly stop landing.

These are the things an experienced duty manager already watches for by hand every morning. They live in production — the clock, real traffic, a partner's behavior — so a lower environment has nothing to show, and the pipeline, which only sees changes, never sees them. The simple rule: a change → your pipeline catches it; live runtime state → someone has to look.

And here's what matters most: the individual signals already exist — each is a native metric or record:

Risk	Native signal that already exists
Integration creep	Lambda Errors / Duration (p99) / Throttles in CloudWatch; the downstream's latency/5xx via AWS X-Ray; Connect ContactFlowErrors
Capacity	AWS/Connect metrics ConcurrentCalls, ConcurrentCallsPercentage, CallsBreachingConcurrencyQuota, CallsPerInterval; Lambda concurrency; Service Quotas
Outbound reachability	per-contact outcome (the destination number + DisconnectDetails) in Amazon Connect contact records
Recording	AWS/Connect metric CallRecordingUploadError; recording fields in contact records

So "only an agent can see these" would be wrong — the individual checks are table stakes. The value is correlating them. Each signal alone looks acceptable on its own dashboard; the risk only appears when you link them across services and against the day's plan — which is what no single-domain tool does:

• Creep × volume. The data-dip error rate has crept from ~1% to ~4% over two weeks — "amber but fine" alone. Add Monday's projected 2× volume and the flow's timeout, and it tips over at peak → one in five authenticated callers fail.
• Three weak signals → one NO-GO. Error-rate amber, capacity "OK," outbound answer-rate slightly down — each acceptable alone. Together: the creep breaks at the campaign's volume, that same volume breaches Lambda concurrency, and the outbound dip means the campaign won't even reach customers. Not ready for Monday.

That linked verdict — ready, or not, and why — is the whole point of what the team built.

The corrective action — and what we nearly built

The root-cause review pointed to a clear corrective action: something had to look across the whole path each morning and connect the signals — the creeping integration, the rising volume, the flow's timeout — before the lines opened.

AWS DevOps Agent was already in place, and it had cut case volume for other scenarios. But it couldn't prevent this one: it's reactive, triggered by an incident — and this failure never raised an incident until it was already too late. So the team weighed the realistic options for the corrective action:

• Keep doing it by hand — a duty manager checking four dashboards every morning. Slow, easy to skip under pressure, and it leaves no record.
• Build it themselves — a scheduled Lambda wiring together CloudWatch, contact records, Service Quotas, and the correlation logic. Doable, but a meaningful amount of undifferentiated code to write, test, and own.

While they were scoping that build, AWS DevOps Agent's custom SRE agents were released — scheduled agents that run on a cadence and reason across signals. It offered exactly the scheduling and cross-service correlation they were about to hand-build, with far less to develop. So they chose it.

The fix: a Start-of-Day Readiness SOP

Contact centers already have a "morning readiness" ritual — someone checks the floor before the lines open. The team automated theirs as a standard operating procedure (SOP) that an AWS DevOps Agent custom agent runs every morning before opening: it gathers the live signals across the path, correlates them, and returns a single go / no-go with reasons, so a human fixes the gap at 7am instead of discovering it at 9. In plain terms, the SOP is the set of skills the agent follows, and the custom agent is what runs it on a schedule.

This maps cleanly to how DevOps Agent works — a custom agent is the same product you already run, used proactively: it executes on a schedule with no incident present, reasons in natural language across the tools you give it, and produces an output. Its skills are essentially runbooks/SOPs the agent follows. And every check here runs on the agent's built-in AWS tools — no custom MCP server to build or host.

The SOP's deliverable is not four independent check results — it is the correlation of them into one "ready / not ready, and why." The per-parameter checks are the inputs; the linked verdict is the output, and the reason the agent exists.

The four readiness parameters — each a dynamic, runtime signal, not a static config check:

SKILLS (modular SKILL.md files, all loaded into ONE custom agent)
┌──────────────────────┬──────────────────────────────────────────────────────────────┐
│ integration-baseline │ data-dip Lambda Errors/Duration(p99)/Throttles + downstream     │
│                      │ latency/5xx (X-Ray) + ContactFlowErrors; creep vs the flow      │
│                      │ timeout at projected volume                                     │
│ capacity-baseline    │ ConcurrentCalls / ConcurrentCallsPercentage / CallsBreaching-   │
│                      │ ConcurrencyQuota + Lambda concurrency vs the next known peak     │
│ reachability-baseline│ per-caller-ID connect/answer outcome from contact records;       │
│                      │ sustained per-number degradation                                │
│ recording-baseline   │ CallRecordingUploadError + recordings landed vs contacts handled │
│ ── correlation ──    │ THE POINT: link all four to each other and to the promise →     │
│                      │ one go/no-go with the reason                                    │
└──────────────────────┴──────────────────────────────────────────────────────────────┘

How to build it

Treat the prompts and expressions as illustrative — adapt them to your estate and verify every metric and API against current docs before you run anything.

Prerequisites: an account with AWS DevOps Agent enabled — eligible AWS Support plans can activate it from the Support Center Console with included credits (see plan eligibility and credits) — and an Agent Space with the contact center accounts connected (read-only) and its topology learned.

1. Create the custom agent (use the Chat method). In the DevOps Agent web app, go to Agents → Custom Agents → Create agent. There are two methods — the Form and Chat — but MCP tools can only be assigned via Chat, and this agent needs tools, so use Chat. You can simply ask: "Create a custom agent that runs our start-of-day contact center readiness check every morning." (Creating a custom agent)

2. Write the system prompt in the documented Goal / Approach / Constraints / Output structure (Writing a system prompt). The Approach is where you tell it to correlate, not just collect:

## Goal
Before the lines open, decide whether the contact center can keep its promise
today — inbound (reach, identify, serve, record) and outbound (calls get through).

## Approach
1. Apply the integration-baseline, capacity-baseline, reachability-baseline and
   recording-baseline skills to gather today's live signals (read-only).
2. THEN correlate them against each other and the next known peak — e.g. an
   error-rate creep that tips past the flow timeout at projected volume; a quota
   the peak will breach; outbound numbers whose connect-rate is degrading.
3. Decide GO or NO-GO for opening, with the specific reason and resource.

## Constraints
- Read-only. Never modify, create, or delete any resource.
- Scope to the in-scope accounts and Region only.
- If a signal can't be determined, say so — do not guess.

## Output
A single JSON go/no-go readiness report; per finding: severity, the services
involved, the correlated evidence, and the recommended action.

3. Assign least-privilege, read-only tools — via Chat. For example: "Add the use_aws and query_cloudwatch_logs tools to the readiness agent." Scope to discovery only (connect:Describe*/List*, cloudwatch:GetMetricData, lambda:Get*/List*, servicequotas:Get*, plus read access to your contact-record stream). Two layers keep it safe: the IAM role you configure, and DevOps Agent's permission guardrail (a session policy applied at assume-role time) — effective permissions are the intersection, so mutating calls are blocked even if a policy allows them. (Configuring tools; Limiting agent access)

4. Author the four skills. Each is a directory whose only mandatory file is a Markdown SKILL.md with frontmatter (name + description written from the agent's perspective — the agent uses it to decide when to load the skill). Keep them in Git and import them, then assign all four. (DevOps Agent Skills) Name the exact signals so the checks are reproducible:

---
name: integration-baseline
description: Use to assess whether the authenticated data dip is degrading under
  load — Lambda errors/latency/throttles and the downstream API's latency/errors,
  against the contact flow's timeout and projected call volume.
---
# integration-baseline
Signals (read-only):
- Lambda Errors, Duration (p99), Throttles  — CloudWatch (AWS/Lambda)
- downstream API latency / 5xx              — AWS X-Ray (or the integration's metrics)
- Connect ContactFlowErrors                 — CloudWatch (AWS/Connect)
Judge: is the error/latency trend creeping toward the flow timeout at the next
peak's projected volume? Report the trend and the projected breach, not a snapshot.

---
name: capacity-baseline
description: Use to assess whether the call path has headroom for the next known
  peak across Connect concurrency and the data-dip Lambda's concurrency.
---
# capacity-baseline
Signals (read-only):
- ConcurrentCalls, ConcurrentCallsPercentage, CallsBreachingConcurrencyQuota,
  CallsPerInterval                          — CloudWatch (AWS/Connect)
- Lambda ConcurrentExecutions, Throttles    — CloudWatch (AWS/Lambda)
- configured limits                         — Service Quotas
Judge: at the next peak's projected volume, which limit breaches first?

---
name: reachability-baseline
description: Use to detect outbound caller-ID numbers whose connect/answer rate is
  degrading (a proxy for carrier scam-labeling), per number, over time.
---
# reachability-baseline
Signals (read-only):
- per-contact outcome from Amazon Connect CONTACT RECORDS (the destination/
  system endpoint + DisconnectDetails) — there is NO per-number CloudWatch metric,
  so derive connect/answer rate per caller-ID number from the contact-record stream.
Judge: which outbound numbers show a sustained drop vs their own baseline?

---
name: recording-baseline
description: Use to detect a compliance-recording capture gap — recordings failing
  to upload, or fewer recordings landing than calls handled.
---
# recording-baseline
Signals (read-only):
- CallRecordingUploadError                  — CloudWatch (AWS/Connect)
- recordings landed vs contacts handled     — recording fields in contact records
                                              vs ContactsHandled
Judge: is the capture rate below the compliance baseline this morning?

5. Validate the agent before you schedule it — this is your pre-production gate. You can't reproduce these production failures in a lower environment, but you can validate the agent's own logic there: on the agent's detail page choose Run Now (read-only / observe-only), open the invocation in History, and watch its trajectory — the ordered reasoning steps, tool calls, and results. Confirm it called the right read-only signals and that its correlation and verdict are sound. Iterate on the prompt and skills until a clean run produces a correct report. Only then proceed to scheduling. (Executing custom agents)

6. Schedule it. On the agent, open the Triggers tab → +, and enter a cron() or rate() expression (a green check confirms validity); pause or delete it later from the same tab:

cron(0 6 ? * MON-FRI *)    # 06:00 UTC, before the lines open. For cron, exactly
                            # one of day-of-month / day-of-week must be '?'.

Schedule syntax follows Amazon EventBridge Scheduler.

7. Read the morning verdict — and keep the trajectory. The agent's output is the Start-of-Day Readiness Verdict defined in the next section; instruct that JSON shape in the system prompt and the agent fills it in. Every run is also recorded as an invocation trajectory — a timestamped, reviewable record of exactly what it checked and concluded. In a regulated contact center that trajectory is itself the answer to an auditor's question, "how did you determine you were ready to open that morning?" — which an ad-hoc human checklist can't produce. (Managing custom agents)

A note on what's a feature, and when

To keep release status precise: AWS DevOps Agent reached general availability in March 2026. Custom agents were added as a capability in June 2026 (no preview label). A separate release-management capability is in preview, US East (N. Virginia) only, in June 2026 — it reviews code changes before deployment and is not used by this readiness agent. Don't conflate the three; confirm current status and Regions before you build. (What's new)

The output: a Start-of-Day Readiness Verdict

The agent produces one clearly-defined deliverable every morning — the Start-of-Day Readiness Verdict — structured so a human can act on it in seconds. It has three parts:

• readiness — the headline call: GO, GO-WITH-WARNINGS, or NO-GO.
• generated — the timestamp of the run.
• findings[] — one entry per correlated risk, each with:
  • severity — CRITICAL / HIGH / MEDIUM.
  • services — the services the finding spans (this is the cross-service correlation, made explicit).
  • correlated_evidence — the linked signals and the projection behind the call.
  • verdict_reason — the business consequence, in plain language.
  • recommended_action — the specific fix to apply before opening.

{
  "readiness": "NO-GO",
  "generated": "2026-06-29T06:00Z",
  "findings": [
    {
      "severity": "CRITICAL",
      "services": ["downstream CRM", "AWS Lambda", "Amazon Connect"],
      "correlated_evidence": "CRM data-dip error rate up 1%→4% over 14 days; Lambda p99 +40%; at the campaign's projected 2x volume this crosses the flow's 8s timeout.",
      "verdict_reason": "Authenticated callers will fail at peak (~9am).",
      "recommended_action": "Raise CRM connection-pool / timeout headroom, or stage the campaign ramp, before opening."
    }
  ]
}

How it's delivered — both forms are native to custom agents, no extra plumbing:

• a readiness artifact (text, a findings table, a trend chart) on the Agent Space Artifacts page; and
• a recommendation per finding on the Improvements page (status Proposed → your team accepts, rejects, or closes it).

How it's used. At ~07:00 the duty manager reads the verdict:

• GO → open the lines with confidence; the trajectory is filed as the record that readiness was checked.
• GO-WITH-WARNINGS → open, with the named non-critical items tracked.
• NO-GO → fix the named gap (or stage the campaign ramp) before 8am — turning a 9am customer-facing incident into a 7am task no customer ever saw.

If you want the verdict where the team already works, DevOps Agent's chat and ticketing integrations can post it to Slack or raise a ticket — optional, and still no custom MCP server required.

What it is, and isn't

• It detects and warns. It never mutates. It recommends; a human acts.
• It is not a replacement for the reactive agent. When something breaks in real time, that's still incident-triggered DevOps Agent's job. The readiness agent hands off to it.
• Cost questions aren't its job — those belong to cost tooling.
• Carrier scam-labeling is external. The agent flags the native connect-rate proxy from your contact records; it can't read carrier labels directly or fix carrier reputation.
• It still uses an LLM's judgment. Pin versions where you can, keep the skills under review, and treat findings as high-quality flags a human confirms.
• Mind inference routing for data residency. Confirm your Region's inference and data-residency behavior in the Supported Regions documentation before deploying — in some Regions, inference for an Agent Space may be routed outside that Region's geography.

How you'd measure whether it worked

This is a design, not a deployed result, so treat measurement as something you instrument, not something this article claims. Take the rate of "silent" production incidents on this contact center — the ones with no deploy and no single-dashboard signal — for the quarter before the SOP, and compare the same class after. The hypothesis is that incidents which used to surface at 9am as customer impact now surface at 7am as findings fixed before opening. Prove it on your own data.

Key takeaway

Your pipeline and your native dashboards are the floor, and a good one — they catch bad changes and they expose every individual signal. What they don't do is look at production on a schedule and correlate those signals across services into the one answer the business actually needs: are we ready to open the lines today? That answer lives in the seams — between a creeping downstream, a rising peak, a degrading number, a missing recording — where no single tool is looking. A scheduled custom agent, composed to your own contact center, can stand in those seams every morning and tell you before the customers do.

---

References

• AWS DevOps Agent — About, Custom agents (overview), What's new
• AWS DevOps Agent — configuration: Creating a custom agent, Executing & scheduling custom agents, Managing custom agents, DevOps Agent Skills, Limiting agent access
• Amazon Connect — Monitoring with CloudWatch (metrics incl. ConcurrentCalls, CallsBreachingConcurrencyQuota, CallRecordingUploadError), Contact records data model
• AWS Lambda CloudWatch metrics · AWS X-Ray · Service Quotas · Amazon EventBridge Scheduler
• Companion articles — Part 1 · Part 2

---

Co-authored by Krish Balaraman, Sr. Enterprise Support Manager, AWS Enterprise Support. (Byline pending confirmation.)