It passed every check and still broke production: shift-left release readiness, from incident to setup

---

⚠️ A note on the scenario and numbers. The bank, the platform team, the application architecture, and every figure, code excerpt, and report here are a composite — representative of patterns seen across several financial-services organizations, not any one customer's environment, and not benchmarks. Code, policies, and review output are illustrative and synthetic. AWS service behaviors and feature status are described from public documentation as of June 2026; the release-management capability described here is in preview, so verify current availability, regional coverage, and feature status against AWS docs before you build.

In the earlier articles in this series, a bank's engineering team put AI operations agents to work against their own support history. In the first article they mined that history for patterns; in the second they scored it against AWS DevOps Agent and deployed it; and in the third they built a scheduled custom agent to catch the runtime failures that nothing ever triggered.

This is about the same bank, a different team, and the opposite blind spot — and it starts on a Saturday. The second half is a complete, doc-anchored walkthrough you can follow to set the same thing up.

Context and purpose

Parts 1–3 covered runtime failures. This one tackles the opposite: change-time failures that pass every pipeline check yet still reach production — because staging drifts from production and tests judge the code, not the change in context. You'll learn to spot that failure class and follow a complete, doc-anchored setup — architecture, validation infrastructure, standards, and a worked pull request — using DevOps Agent's release-readiness review (preview) to catch it before merge.

The Saturday release that wasn't

It was a Saturday evening, and the bank's platform team was shipping the safe kind of release — small change, fully reviewed, every pipeline stage green. The release lead had done this a hundred times.

By the time anyone noticed, a database-administration tool the team used to reach a managed database was returning gateway timeouts. The application layer was up. The host was up. The database was up. Everything was "up," and nothing worked.

The root cause, found two hours later on a screen-share, was almost insultingly small. The staging launch template referenced a release-candidate base operating-system image instead of the production-pinned build the team had standardized on. The environment came up on the wrong image, a library mismatch broke the connector, and the tool couldn't talk to the database.

Here is what made the release lead put her coffee down. Nothing in the pipeline was broken. The code compiled. The tests passed. The change was reviewed and approved. The pipeline did its job perfectly — and shipped a failure straight into the environment, because the failure didn't live in the code. It lived in the difference between what the team intended and what the change actually referenced.

Three in one quarter

What unsettled her wasn't the outage by itself. It was that she'd seen this shape before — twice that same quarter.

A few weeks earlier, a different change had deployed cleanly and then served nothing but 502 and 504 gateway timeouts. Each component — application, host, database — was individually healthy; the path between them was not. The build had pointed the connector at the database over a route that resolved in staging but not in the target environment, so the service came up clean and then timed out on every call. Green deploy. Broken behavior.

A month before that, the scariest one — scary precisely because nothing fell over. A CloudFormation change added a service role, and to get a deployment unblocked late in the day the policy was written broader than it needed to be: a wildcard action on a sensitive resource and a permissive iam:PassRole. It compiled, it deployed, everything worked. Nobody noticed the change had quietly widened what anything assuming that role could do — until the security lead found it weeks later, reading diffs.

Three releases. Three outcomes from embarrassing to dangerous. Not one was a code defect; every one passed every check the pipeline had. The release lead wrote the common thread on a whiteboard: these weren't bugs in the code — they were mismatches between the change and the world it was about to enter. A test suite can't catch those, because a test suite tests the code, not the change against the standard, the environment, the access model, or a real run.

It was the same lesson as the runtime agent in the previous article, pointed the other way. There, failures rotted in after deploy and the fix watched production. Here, they were wrong the moment they shipped, and the fix had to live earlier — on the pull request, before merge.

The turn: a reviewer that reads the change in context

What the team wanted was a reviewer that read each change the way an experienced engineer would, if that engineer had perfect memory of the team's standards, its dependency graph, and how the system ran in production.

That is what the release-readiness review in AWS DevOps Agent is built to do. DevOps Agent's post-deployment operations are generally available; the release-management capabilities — including the release-readiness review — are currently in preview, in the US East (N. Virginia) Region, at no additional cost (What's New). The docs describe release management as "the automated validation layer between code generation and production deployment" (user guide). It reasons over the whole change — standards the team writes in plain English, the cross-repository dependency graph, access-control changes, and a real build-and-run — and returns findings as comments on the pull request, in the console, in the IDE, and on demand through chat (features).

The rest of this article is the team turning that into a working setup — the architecture they run, the validation infrastructure they stood up, and one real pull request walked end to end.

The application

The team owns account-service, an internal API other bank systems call to look up customer account details. It's a composite of a very common financial-services shape:

ACCOUNT-SERVICE  (request paths)
  internal callers  -->  API Gateway  -->  Lambda (account-service)  -->  RDS for PostgreSQL (Multi-AZ)
  ops engineers     -->  DB-admin tool on Amazon EC2                 -->  RDS for PostgreSQL
                         (the EC2 base image is set by a launch template)

REPOSITORIES
  account-service (application)   --depends on-->   platform-infra (CloudFormation)
  platform-infra defines:  VPC, private subnets, security groups, RDS,
                           EC2 launch template   [where the Saturday drift lived],
                           IAM roles             [where the access scare lived]

DELIVERY
  GitHub  --pull request-->  release-readiness review (runs on every PR)

CROSS-REPO EDGE
  account-service consumes platform-infra outputs:  DB security-group ID, DB endpoint

Two repositories, one dependency edge, all delivered through GitHub. That edge — account-service depending on platform-infra outputs — is exactly the kind of relationship a single-repo test never sees, and it's where the review's dependency graph earns its place.

Setting up the validation infrastructure

This is the part that does the real-time validation: when a pull request opens, DevOps Agent stands up an AWS-managed verification environment, clones the change into it on dedicated compute with restricted networking, installs the build tools it detects, builds and runs the change, and evaluates it against your standards, dependencies, and access controls. Here's how the team wired it. The preview runs only in US East (N. Virginia) (us-east-1), so all of this is in a us-east-1 Agent Space.

1. Connect the repositories. In the DevOps Agent console, open the Agent Space, go to the Capabilities tab, choose Add under Pipeline, select GitHub, and complete the OAuth flow to install the AWS DevOps Agent app — granting it access to account-service and platform-infra. Then connect both repositories to the Agent Space under the Pipeline section. The agent indexes them and builds the cross-repository dependency knowledge graph (this takes a little while, so early reviews sharpen as indexing completes). Steps: Connecting GitHub · Connecting GitLab.

2. Turn the review on per repository. In the GitHub integration settings, open the Code Review and Automated Testing section and enable the two per-repository capabilities (both on by default): Auto trigger change review — runs a release-readiness review on every PR — and Automated verification testing — builds, runs, and tests the change in the managed verification environment. See the Getting started section of Release readiness code reviews. (A fuller mode — autonomous release testing — goes further for web and API applications, generating change-specific test plans and running them against a deployed instance in a customer-provisioned environment rather than only the managed verification environment; it needs additional setup and is beyond this walkthrough's scope.)

3. Give the verification environment access to private dependencies (the real-time validation infra). account-service builds against an internal package registry and needs the RDS endpoint to run its functional checks — neither is on the public internet. By default the verification environment is network-isolated with an outbound allowlist (it can already reach AWS services, *.public.ecr.aws, Docker, GitHub/GitLab, and the common language registries — npm, PyPI, Maven, and so on; the current allowlist is in the user guide). To reach private systems you attach it to your own VPC:

• Create a private connection that targets the VPC where the registry and database are reachable (Console or CLI). The verification environment provisions and manages an elastic network interface (ENI) inside that VPC, so traffic follows your own routes and firewall rules — your code and artifacts never leave your network boundary during the review.
• In Capabilities → GitHub → Code Review and Automated Testing, associate that private connection, and select a Runtime role — an IAM role DevOps Agent assumes during builds to reach internal services. It must allow access to AWS Secrets Manager in the same account (for the registry and database credentials), and AWS recommends a role distinct from your primary agent role. Then Save.

An illustrative least-privilege runtime role (synthetic — scope to your own resources):

# Runtime role assumed by the verification environment during release-readiness builds
RuntimeRolePolicy:
  Version: "2012-10-17"
  Statement:
    - Sid: ReadBuildAndDbSecrets
      Effect: Allow
      Action: secretsmanager:GetSecretValue
      Resource:
        - arn:aws:secretsmanager:us-east-1:111111111111:secret:account-service/registry-*
        - arn:aws:secretsmanager:us-east-1:111111111111:secret:account-service/rds-readonly-*
    - Sid: EcrAuth                       # GetAuthorizationToken does not support
      Effect: Allow                       # resource-level scoping; "*" is required here
      Action: ecr:GetAuthorizationToken
      Resource: "*"
    - Sid: PullPrivateImages              # these DO support resource scoping
      Effect: Allow
      Action:
        - ecr:BatchGetImage
        - ecr:GetDownloadUrlForLayer
      Resource: arn:aws:ecr:us-east-1:111111111111:repository/account-service/*

Setup details are in the Configure private VPC access section of Release readiness code reviews. The underlying mechanism is AWS DevOps Agent private connections — the same VPC-attachment capability used for its other private integrations — explained in Securely connect AWS DevOps Agent to private services in your VPCs on the AWS DevOps Blog.

4. (Optional) tune how the build runs. If the agent needs hints — the exact build command, what counts as a passing build, which user journeys to exercise — you provide them in an AGENTS.md agent-instructions file in the repository. An illustrative example:

# AGENTS.md — account-service

## Build
Run `make build`. A passing build compiles and produces the Lambda bundle with no errors.

## Verification testing
- Exercise the GET /accounts/{id} happy path and the 404 (unknown id) path.
- Treat any 5xx response during the user-journey check as a failure.
- Do not run load or soak tests in the verification environment.

Writing the standards

The "policies" the review enforces are standards written in plain English as natural-language instructions, in the console under Knowledge → Instructions → Release readiness review (choose All agents to apply space-wide). With none defined, the agent applies general best practices. The team encoded the three lessons from their bad quarter as an illustrative standard set:

# account-service / platform-infra — release readiness standards

1. Base image (Blocking): Infrastructure changes MUST reference the
   production-pinned base image (the AMI tagged pinned=true). Flag any
   release-candidate or unpinned image.

2. IAM least privilege (Warning): IAM policies MUST NOT use wildcard actions
   ("*") on sensitive resources, and MUST NOT grant iam:PassRole without a
   resource scope.

3. Network exposure (Blocking): Security groups MUST NOT open 0.0.0.0/0
   ingress on any port. Ingress MUST be scoped to approved CIDRs or security
   groups; internet exposure is allowed only via the approved ALB/WAF tier.

4. Functional gate (Blocking): API and Lambda changes MUST build, run, and
   pass user-journey checks in the verification environment before merge.

One pull request, end to end

A developer opens a routine pull request on platform-infra: refresh the DB-admin host's base image and add an IAM role for a new nightly batch job. The diff (illustrative):

  # platform-infra/templates/db-admin-host.yaml
  DbAdminLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateData:
-       ImageId: ami-0prod1111pinned   # base-os 8, pinned=true
+       ImageId: ami-0rc9999candidate  # base-os 9, release candidate

  # platform-infra/templates/batch-role.yaml
+ BatchJobRole:
+   Type: AWS::IAM::Role
+   Properties:
+     Policies:
+       - PolicyName: batch-access
+         PolicyDocument:
+           Statement:
+             - Effect: Allow
+               Action: "*"                       # broad
+               Resource: "*"
+             - Effect: Allow
+               Action: iam:PassRole              # unscoped
+               Resource: "*"

Opening the PR triggers the review (the team also wired it as a required status check — in GitHub, Settings → Branches → Branch protection rules → Require status checks to pass, selecting the DevOps Agent check — so a Blocking finding stops the merge). Within a few minutes the agent has: indexed the change, built and run account-service against the change in the verification environment (reaching the private registry and a read-only database through the VPC connection), checked the cross-repo edge to confirm the security-group output account-service consumes is unchanged, and evaluated the diff against the four standards.

The release-readiness report leads with one overall recommended action — BLOCK, Proceed with Caution, or Safe to Release — then a changes summary, a risk analysis with affected code locations, and recommended fixes; each finding carries a severity (Blocking, Warning, or Informational), and an execution journal records every step the agent took. The report on this PR (illustrative):

Finding	Location	Severity
Base image is a release candidate, not the pinned=true build (Standard 1)	db-admin-host.yaml:7	Blocking
IAM policy uses wildcard action and unscoped iam:PassRole (Standard 2)	batch-role.yaml:9	Warning
account-service builds and passes user-journey checks	verification env	Informational
No breaking change to the security-group output consumed by account-service	dependency graph	Informational

Overall recommended action: BLOCK — driven by the Standard 1 violation. Posted as inline comments on the two lines, with the overall status on the PR.

The developer fixes both in the same PR — pin the production image, and scope the policy:

- ImageId: ami-0rc9999candidate
+ ImageId: ami-0prod1111pinned

- Action: "*"
- Resource: "*"
+ Action:
+   - s3:GetObject
+   - s3:PutObject
+ Resource: arn:aws:s3:::account-service-batch-ledger/*
- Action: iam:PassRole
- Resource: "*"
+ Action: iam:PassRole
+ Resource: arn:aws:iam::111111111111:role/batch-execution-role
+ Condition:
+   StringEquals:
+     iam:PassedToService: ecs-tasks.amazonaws.com

The push re-triggers the review. The base image now matches the pinned standard, the policy is scoped, the functional check still passes, and the dependency edge is clean — recommended action: Safe to Release. The required status check goes green and the change merges. Total elapsed time: minutes, on the pull request — not a two-hour screen-share at the weekend.

If a developer would rather catch this even earlier, the same review runs against in-progress changes in the IDE through the DevOps Agent Kiro Power or the Claude Code plugin, before the code is ever committed.

Where the line is

The release lead was careful, in the retro, to say what the review didn't do — a tool that quietly overpromises is one people stop reading:

• It is not a replacement for tests, linters, or human review. It's a layer that reasons about the change in context.
• The access-control check evaluates infrastructure-as-code changes — for CloudFormation, it verifies that IAM policies, resource policies, and network configurations follow Well-Architected best practices. It does not evaluate in-cluster controls such as Kubernetes RBAC or service-account-token and certificate-trust settings; an in-cluster trust failure would, at best, surface through functional verification.
• It does not cover everything else either. Capacity headroom under peak load and single-Availability-Zone resilience gaps are real and preventable — but they're runtime and architecture-review concerns (the previous article, and a Well-Architected review), not change-time checks.
• It is in preview, in one Region, and will evolve. Re-check the docs before relying on any specific behavior.

What the quarter taught them

The runtime agent in the previous article caught the failures the lower environment couldn't see in time. The release-readiness review caught the ones it couldn't see at all — because staging had drifted from production, because the suite tested the code instead of the change, or because a grant was valid and too broad and no test was looking.

Both are the same goal from two directions: prevent the preventable. A green pipeline told the team the code was correct. It never told them the change was ready. Moving that second judgment left — onto the pull request, before merge — is what turned the next bad Saturday into a comment nobody had to lose a weekend over.

---

References

• AWS DevOps Agent — About AWS DevOps Agent
• AWS DevOps Agent — feature overview
• AWS DevOps Agent release management (preview) — What's New
• AWS News Blog — release management capabilities to assess code changes before production (preview)
• AWS DevOps Blog — Securely connect AWS DevOps Agent to private services in your VPCs (private-connections architecture)
• AWS DevOps Agent User Guide — release management overview
• AWS DevOps Agent User Guide — release readiness code reviews
• AWS Well-Architected Framework

---

By Kiran Khambete and Krish Balaraman, AWS Enterprise Support.