Deploying AWS DevOps Agent in Multi-Account Enterprise Environments: A Strategy for Cross-Account Root Cause Analysis

Introduction

Enterprise organizations running 50–1000+ AWS accounts face a fundamental challenge with incident response: a single production incident rarely stays within one account. A latency spike in your API Gateway (Account A) might trace to a database connection pool exhaustion in RDS (Account B), triggered by a deployment pushed through a CI/CD pipeline in your tooling account (Account C).

AWS DevOps Agent is an autonomous AI-powered investigation agent that can correlate signals across multiple AWS accounts, reason through competing hypotheses, and identify root causes — often before your on-call engineer opens their laptop.

This article covers the deployment strategy, architecture patterns, and best practices for deploying AWS DevOps Agent across a multi-account Enterprise landscape so the agent can effectively find root causes across your distributed infrastructure.

---

How AWS DevOps Agent Finds Root Causes

Before diving into multi-account deployment, it's important to understand the agent's investigation methodology:

Multi-Agent Reasoning Architecture

AWS DevOps Agent uses a structured lifecycle with specialized capabilities:

1. Topology Graph (Foundation) — Auto-discovers resources and relationships across all connected accounts. This is the "map" the agent navigates during investigations.
2. Triage — Correlates incoming alerts with related alarms across services and accounts. Reduces noise by grouping related signals into a single investigation.
3. Investigation — Deep multi-hypothesis root cause analysis:
   • Generates multiple competing theories simultaneously
   • Validates each with both supporting AND counter-evidence
   • Converges on root cause only when evidence conclusively supports it
4. Mitigation — Generates remediation plans with rollback procedures (read-only; does not execute).
5. Prevention — Analyzes patterns across historical incidents to identify systemic improvements.

Why Topology Matters for Cross-Account RCA

The topology graph is what enables cross-account root cause analysis. Without it, the agent would search blindly through telemetry. With it, the agent:

• Follows dependency chains across account boundaries
• Assesses blast radius of a failing component
• Correlates deployment timelines from CI/CD accounts with production symptoms
• Identifies upstream/downstream impact through mapped relationships

The topology is built through:

• CloudFormation stack analysis (including CDK-synthesized stacks)
• Tag-based discovery via AWS Resource Explorer
• Behavioral mapping through CloudWatch Application Signals, Datadog, Dynatrace
• CI/CD pipeline integration linking resources to deployment processes

---

Multi-Account Architecture: Key Concepts

Mapping to AWS Landing Zone / Control Tower Account Structure

Most enterprises with 50–1000 accounts provision their environment via AWS Control Tower or a custom Landing Zone solution. The resulting OU and account structure typically looks like this:

AWS Organization (Management Account)
│
├── Security OU
│   ├── Log Archive Account          ← Centralized CloudTrail, Config, VPC Flow Logs
│   ├── Audit Account                ← Security Hub, GuardDuty delegated admin
│   └── Security Tooling Account     ← SIEM, forensics, Security Lake
│
├── Infrastructure OU
│   ├── Network Hub Account          ← Transit Gateway, Route 53, shared VPCs
│   ├── Shared Services Account      ← Active Directory, DNS, common tooling
│   └── CI/CD Tooling Account        ← CodePipeline, GitHub Actions runners
│
├── Sandbox OU
│   └── Developer sandbox accounts
│
├── Workloads OU
│   ├── BU-A OU
│   │   ├── bu-a-prod (Production)
│   │   ├── bu-a-staging
│   │   └── bu-a-dev
│   │
│   ├── BU-B OU
│   │   ├── bu-b-prod (Production)
│   │   ├── bu-b-staging
│   │   └── bu-b-dev
│   │
│   └── BU-C OU (100+ accounts)
│       ├── bu-c-prod-app1
│       ├── bu-c-prod-app2
│       └── ...
│
└── Suspended OU
    └── Decommissioned accounts

Where Does the DevOps Agent Fit in This Structure?

The critical question is: where do you create the Agent Space (primary account), and which accounts become secondaries?

Recommended: Dedicated DevOps Agent Account Under Infrastructure OU

├── Infrastructure OU
│   ├── Network Hub Account
│   ├── Shared Services Account
│   ├── CI/CD Tooling Account
│   └── DevOps Agent Account (NEW)    ← Agent Spaces live here
│       ├── Agent Space: "BU-A Payments"
│       ├── Agent Space: "BU-B Orders"
│       └── Agent Space: "Platform Networking"

Why a separate account?

• Isolated from workload blast radius — workload-level SCPs don't affect the Agent
• Centralized IAM governance for all Agent Spaces
• Independent from Log Archive (which has restrictive write-deny policies)
• Clear cost allocation via dedicated billing

Role of Centralized Accounts in DevOps Agent Investigations

Log Archive Account — The Investigation Goldmine

Your centralized Log Archive account holds CloudTrail logs, AWS Config history, and VPC Flow Logs from ALL accounts in the organization. Adding it as a secondary account gives the agent:

• Organization-wide CloudTrail — The agent can trace API calls across any account without needing secondary access to every individual account
• AWS Config history — Configuration timeline changes across the fleet
• VPC Flow Logs — Network-level traffic patterns for connectivity issues

⚠️ Important: Log Archive accounts typically have restrictive SCPs (deny all writes). The DevOps Agent IAM role only needs AIDevOpsAgentAccessPolicy (read-only), which is compatible with these restrictions.

# Log Archive secondary account — read-only role is compatible with deny-write SCPs
DevOpsAgentRole:
  ManagedPolicyArns:
    - arn:aws:iam::aws:policy/AIDevOpsAgentAccessPolicy
  # Additional inline for CloudWatch Logs cross-account read
  Policies:
    - PolicyName: LogArchiveAccess
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - logs:FilterLogEvents
              - logs:GetLogEvents
              - logs:DescribeLogGroups
              - logs:DescribeLogStreams
            Resource: '*'

Network Hub Account — Tracing Connectivity Root Causes

Network-related incidents frequently originate in the Network Hub account (Transit Gateway misconfigurations, route table changes, Security Group modifications). Adding this as a secondary account enables:

• Transit Gateway route analysis — The agent can check if routes between VPCs/accounts were modified
• VPC peering / PrivateLink status — Verify inter-account connectivity
• DNS resolution — Route 53 Resolver rule changes that break service discovery
• Firewall rules — AWS Network Firewall policy changes

Investigation Example:
  Symptom: "Service in BU-A-Prod can't reach database in BU-B-Prod"
  Agent traces:
    1. BU-A-Prod account → Security Group ✅ (allows outbound)
    2. Network Hub account → Transit Gateway route table ❌
       (Route to BU-B VPC CIDR was removed 15 min ago by user X)
    3. Root Cause: Unintended TGW route deletion during infra change

Security Tooling / Security Lake Account

For organizations using AWS Security Lake (centralized security data in OCSF format):

• Add the Security Lake account as a secondary account
• Pre-create Athena tables pointing to Security Lake S3 buckets
• The agent can correlate security findings with operational incidents
• Especially powerful for "was this operational issue caused by a security event?"

---

Multi-BU Deployment Scenarios

Scenario 1: Centralized Operations — Shared Core Components

Pattern: A central Cloud Platform/SRE team manages networking, logging, and security for all BUs. Each BU manages their own workload accounts but relies on shared infrastructure.

┌──────────────────────────────────────────────────────────────────────┐
│                  DevOps Agent Account (Infrastructure OU)             │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────────┐  ┌────────────────────┐  ┌──────────────┐  │
│  │ Agent Space:        │  │ Agent Space:       │  │ Agent Space: │  │
│  │ "Platform Network"  │  │ "BU-A Commerce"    │  │ "BU-B Data"  │  │
│  │                     │  │                    │  │              │  │
│  │ Secondaries:        │  │ Secondaries:       │  │ Secondaries: │  │
│  │ • Network Hub       │  │ • bu-a-prod        │  │ • bu-b-prod  │  │
│  │ • Log Archive       │  │ • bu-a-staging     │  │ • bu-b-prod2 │  │
│  │ • All BU Prod accts │  │ • Network Hub      │  │ • Log Archive│  │
│  │   (connectivity)    │  │ • Log Archive      │  │ • Network Hub│  │
│  │                     │  │ • CI/CD Tooling    │  │ • Shared Svcs│  │
│  └─────────────────────┘  └────────────────────┘  └──────────────┘  │
└──────────────────────────────────────────────────────────────────────┘

Key insight: Each BU's Agent Space should include the Network Hub and Log Archive as secondary accounts, because:

• The agent needs to check if a BU-specific incident was caused by a change in shared infrastructure
• Cross-BU traffic flows through Transit Gateway in the Network Hub
• Centralized CloudTrail in Log Archive provides organization-wide visibility

Scenario 2: Federated Operations — BUs Manage Their Own Core

Pattern: Each BU operates independently with their own networking, logging, and CI/CD accounts. Minimal shared infrastructure at the org level.

┌──────────────────────────────────────────────────────────────────────┐
│                  DevOps Agent Account (Infrastructure OU)             │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────────────────┐     ┌─────────────────────────┐        │
│  │ Agent Space: "BU-A"      │     │ Agent Space: "BU-B"      │        │
│  │                          │     │                          │        │
│  │ Secondaries:             │     │ Secondaries:             │        │
│  │ • bu-a-prod              │     │ • bu-b-prod              │        │
│  │ • bu-a-staging           │     │ • bu-b-staging           │        │
│  │ • bu-a-network (own TGW) │     │ • bu-b-network (own TGW) │        │
│  │ • bu-a-logs (own logs)   │     │ • bu-b-logs (own logs)   │        │
│  │ • bu-a-cicd              │     │ • bu-b-cicd              │        │
│  │                          │     │                          │        │
│  │ NO cross-BU access       │     │ NO cross-BU access       │        │
│  └─────────────────────────┘     └─────────────────────────┘        │
│                                                                      │
│  ┌─────────────────────────┐                                         │
│  │ Agent Space: "Org Core"  │  ← Only org-level shared infra         │
│  │ Secondaries:             │                                         │
│  │ • Org Log Archive        │                                         │
│  │ • Org Transit Gateway    │                                         │
│  │ • Security Hub account   │                                         │
│  └─────────────────────────┘                                         │
└──────────────────────────────────────────────────────────────────────┘

Key insight: In federated models, BU isolation is critical. BU-A's Agent Space should NOT have secondary access to BU-B's accounts. This ensures:

• Data segregation between business units
• Compliance with internal data classification policies
• Each BU team only sees their own investigations

Scenario 3: Hybrid — Shared Network/Logs + Independent BU Workloads

Pattern (Most Common): Central team manages networking and logging; BUs manage their own workloads and CI/CD but share the network backbone.

┌────────────────────────────────────────────────────────────────┐
│               DevOps Agent Account                              │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│  Per-BU Agent Spaces:                                          │
│  ┌──────────────────┐  ┌──────────────────┐                   │
│  │ "BU-A Payments"  │  │ "BU-B Analytics" │  ...              │
│  │ Secondaries:     │  │ Secondaries:     │                   │
│  │ • bu-a-prod      │  │ • bu-b-prod      │                   │
│  │ • bu-a-staging   │  │ • bu-b-prod-2    │                   │
│  │ • bu-a-cicd      │  │ • bu-b-cicd      │                   │
│  │ • Network Hub ←──┼──┼── Shared         │                   │
│  │ • Log Archive ←──┼──┼── Shared         │                   │
│  └──────────────────┘  └──────────────────┘                   │
│                                                                │
│  Platform Agent Space (Central SRE owns):                      │
│  ┌──────────────────────────────────────────┐                  │
│  │ "Platform Infrastructure"                 │                  │
│  │ Secondaries:                              │                  │
│  │ • Network Hub (TGW, R53, Firewall)        │                  │
│  │ • Log Archive (Org-wide CloudTrail)       │                  │
│  │ • Security Tooling (GuardDuty, SecHub)    │                  │
│  │ • ALL BU Prod accounts (for network RCA)  │                  │
│  └──────────────────────────────────────────┘                  │
└────────────────────────────────────────────────────────────────┘

Key insight: The Network Hub and Log Archive are added as secondaries to BOTH:

1. Each BU's Agent Space — so BU investigations can trace into shared infra
2. The Platform Agent Space — so the central SRE team can investigate network/platform issues that affect multiple BUs

This is NOT a security concern because the DevOps Agent role is read-only. The agent cannot modify resources in shared accounts — it can only read telemetry and configuration for investigation purposes.

Agent Space — The Logical Container

An Agent Space is the central unit of deployment. It:

• Maintains its own topology graph
• Has isolated investigation history
• Contains its own integrations and skills
• Scopes what the agent can see and do

Each Agent Space has:

• One primary account (where the Agent Space is created)
• Multiple secondary accounts (added for cross-account visibility)

Primary vs. Secondary Accounts

Aspect	Primary Account	Secondary Accounts
Purpose	Hosts the Agent Space infrastructure	Provides visibility into workload resources
IAM Role	DevOpsAgentRole-AgentSpace	DevOpsAgentRole-SecondaryAccount
Policy	AIDevOpsAgentAccessPolicy + inline	AIDevOpsAgentAccessPolicy + inline
Trust	aidevops.amazonaws.com service principal	Trusts Agent Space ARN in primary account
Count	1 per Agent Space	Many (no hard limit; stress-tested at hundreds)

---

Deployment Strategy for 50–1000 Account Enterprises

Strategy 1: Dedicated Monitoring Account (Recommended)

┌─────────────────────────────────────────────────┐
│          Monitoring Account (Primary)            │
│  ┌───────────────┐  ┌───────────────┐           │
│  │ Agent Space A │  │ Agent Space B │  ...       │
│  │ (Service X)   │  │ (Service Y)   │           │
│  └───────┬───────┘  └───────┬───────┘           │
└──────────┼──────────────────┼───────────────────┘
           │                  │
    ┌──────┴──────┐    ┌──────┴──────┐
    │ Secondary   │    │ Secondary   │
    │ Accounts    │    │ Accounts    │
    │ (Prod, Dev, │    │ (Prod, Dev, │
    │  Staging)   │    │  Staging)   │
    └─────────────┘    └─────────────┘

Why?

• Centralizes operational tooling away from workload accounts
• Follows AWS multi-account best practices (separate operations from workloads)
• Simplifies IAM governance — security team controls one account
• Enables independent lifecycle management of the monitoring infrastructure

Strategy 2: One Agent Space per Logical Service

For enterprises with 50–1000 accounts, the recommended pattern is:

One Agent Space per logical service (or tightly-coupled service group)

Why per-service isolation?

• Scoped investigations — The agent can access ALL accounts/resources in an Agent Space during ANY investigation. Smaller topology = tighter reasoning and better root-cause accuracy.
• Least-privilege IAM — Each Agent Space's role only accesses what that service needs.
• Reduced noise — Alert associations (alarm filters, resolver groups) are scoped per Agent Space, so unrelated alarms don't trigger investigations.
• Ownership transfer — If a service changes teams, hand off the entire Agent Space cleanly.

Strategy 3: Account Grouping by Environment

┌────────────────────────────────────┐
│     Agent Space: "Service X"        │
├────────────────────────────────────┤
│ Primary: Monitoring Account         │
│ Secondary: Prod Account (us-east-1) │
│ Secondary: Prod Account (eu-west-1) │
│ Secondary: CI/CD Tooling Account    │
│ Secondary: Shared Services Account  │
└────────────────────────────────────┘

Best practice: Separate pre-production from production Agent Spaces. This is the only universally confirmed structuring guidance from the service team.

Strategy 4: OU-Aligned Deployment with StackSets (Landing Zone Native)

For organizations using Control Tower / Landing Zone with the OU structure above, deploy the DevOps Agent secondary role per-OU using StackSets with auto-deployment:

StackSet: "DevOpsAgent-BU-A-Role"
  → Target: ou-workloads-bu-a
  → Auto-deploy: Enabled (new accounts get the role automatically)
  → Agent Space ARN: points to BU-A's Agent Space

StackSet: "DevOpsAgent-BU-B-Role"
  → Target: ou-workloads-bu-b
  → Auto-deploy: Enabled
  → Agent Space ARN: points to BU-B's Agent Space

StackSet: "DevOpsAgent-Platform-Role"
  → Target: ou-infrastructure + ou-security
  → Auto-deploy: Enabled
  → Agent Space ARN: points to Platform Agent Space

Why OU-aligned StackSets?

• New accounts provisioned via Account Factory/vending machine automatically get the DevOps Agent role — zero manual onboarding
• OU-level targeting ensures BU isolation (BU-A role trusts BU-A's Agent Space only)
• Central team manages the StackSet; BU teams don't need to worry about it
• Removing an account from the OU automatically removes the role (when RetainStacksOnAccountRemoval=false)

Handling the "Shared Accounts" Problem

When a single account (e.g., Network Hub) needs to be a secondary in multiple Agent Spaces, you need multiple IAM roles with different names:

# Network Hub Account — multiple DevOps Agent roles

# Role for BU-A's Agent Space
DevOpsAgentRole-BU-A:
  Trust: Agent Space ARN of BU-A
  Policy: AIDevOpsAgentAccessPolicy + inline

# Role for BU-B's Agent Space
DevOpsAgentRole-BU-B:
  Trust: Agent Space ARN of BU-B
  Policy: AIDevOpsAgentAccessPolicy + inline

# Role for Platform Agent Space
DevOpsAgentRole-Platform:
  Trust: Agent Space ARN of Platform
  Policy: AIDevOpsAgentAccessPolicy + inline

⚠️ Critical pitfall: DevOps Agent CDK/CloudFormation samples use a fixed role name (DevOpsAgentRole-SecondaryAccount). If multiple Agent Spaces need secondary access to the same account, you must use unique role names per Agent Space. Otherwise, the second deployment overwrites the trust policy of the first.

SCP Considerations for Landing Zone Environments

Common Landing Zone SCPs that can impact DevOps Agent:

SCP Pattern	Impact on DevOps Agent	Mitigation
Region deny (aws:RequestedRegion)	Agent cannot investigate resources in denied regions	Add an exception for the DevOps Agent role ARN in the SCP condition
Deny creating IAM roles	Cannot deploy the DevOpsAgentRole	Deploy role before applying SCP, or add exception for CloudFormation StackSets service principal
Deny all writes (Log Archive)	No impact — DevOps Agent role is read-only	None needed
Deny VPC changes (guardrail)	No impact — Agent doesn't modify resources	None needed
Require tags on resources	May block Resource Explorer SLR creation	Add exception for iam:CreateServiceLinkedRole

SCP exception for DevOps Agent (region restriction example):

{
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "StringNotEquals": {
      "aws:RequestedRegion": ["us-east-1", "eu-west-1"]
    },
    "ArnNotLike": {
      "aws:PrincipalArn": "arn:aws:iam::*:role/DevOpsAgentRole-*"
    }
  }
}

Scaling Guidance

Account Range	Recommended Approach
50–100 accounts	5–15 Agent Spaces (per-service grouping)
100–400 accounts	10–40 Agent Spaces; consider regional isolation
400–1000+ accounts	Open a SpecReq for tailored architecture guidance

Note: The service team confirmed no performance issues in stress tests with hundreds of secondary accounts per Agent Space. Skills limit is 200 per Agent Space.

---

Step-by-Step Deployment for Cross-Account Setup

Prerequisites

• AWS DevOps Agent is available in: US East (N. Virginia), US West (Oregon), Canada (Central), South America (São Paulo), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), Europe (London)
• Administrative access to both primary and secondary accounts
• IAM permissions to create roles

Option A: Infrastructure-as-Code with AWS CDK (Recommended for Enterprise)

# Clone the sample CDK app
git clone https://github.com/aws-samples/sample-aws-devops-agent-cdk.git
cd sample-aws-devops-agent-cdk
npm install

Part 1 — Deploy Agent Space (Monitoring Account):

// lib/constants.ts
export const MONITORING_ACCOUNT_ID = "111111111111";

cdk bootstrap aws://111111111111/us-east-1 --profile monitoring
npm run build
cdk deploy DevOpsAgentStack --profile monitoring

Part 2 — Deploy Cross-Account Role (Each Secondary Account):

// lib/constants.ts
export const SERVICE_ACCOUNT_ID = "222222222222";
export const AGENT_SPACE_ARN = "arn:aws:aidevops:us-east-1:111111111111:agentspace/abc123";

cdk bootstrap aws://222222222222/us-east-1 --profile service
cdk deploy ServiceStack --profile service

Option B: CloudFormation StackSets (For Bulk Deployment)

For enterprises with 100+ accounts, use CloudFormation StackSets to deploy the secondary account IAM role across all accounts in an Organizational Unit (OU):

# DevOpsAgentSecondaryRole.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'DevOps Agent Secondary Account Role'

Parameters:
  AgentSpaceArn:
    Type: String
    Description: 'ARN of the Agent Space in the monitoring account'
  RoleName:
    Type: String
    Default: 'DevOpsAgentRole-SecondaryAccount'

Resources:
  DevOpsAgentRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: !Ref RoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: aidevops.amazonaws.com
            Action: 'sts:AssumeRole'
            Condition:
              StringEquals:
                aws:SourceArn: !Ref AgentSpaceArn
                aws:SourceAccount: !Select [4, !Split [':', !Ref AgentSpaceArn]]
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AIDevOpsAgentAccessPolicy'
      Policies:
        - PolicyName: DevOpsAgentInlinePolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'iam:CreateServiceLinkedRole'
                Resource: '*'
                Condition:
                  StringEquals:
                    iam:AWSServiceName: resource-explorer-2.amazonaws.com

Deploy via StackSets:

aws cloudformation create-stack-set \
  --stack-set-name DevOpsAgentSecondaryRoles \
  --template-body file://DevOpsAgentSecondaryRole.yaml \
  --parameters ParameterKey=AgentSpaceArn,ParameterValue=arn:aws:aidevops:us-east-1:111111111111:agentspace/abc123 \
  --permission-model SERVICE_MANAGED \
  --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false

aws cloudformation create-stack-instances \
  --stack-set-name DevOpsAgentSecondaryRoles \
  --deployment-targets OrganizationalUnitIds=ou-OrgUnits \
  --regions us-east-1

Option C: Terraform

AWS also provides a Terraform module that deploys the Agent Space and cross-account roles.

---

Cross-Account RCA Strategy: How the Agent Traces Through Landing Zone Layers

Understanding how the agent navigates your Landing Zone structure during an investigation is key to proper secondary account configuration.

Investigation Flow: BU Workload → Shared Network → Centralized Logs

Example: E-commerce checkout latency spike (BU-A Prod account)

Step 1: Alert fires in BU-A Prod account (CloudWatch Alarm)
        → Agent checks: BU-A workload metrics, ECS task health, ALB 5xx

Step 2: Agent identifies downstream RDS connection timeouts
        → Traces to: Security Group allows traffic, RDS instance healthy
        → Hypothesis: Network path issue

Step 3: Agent checks Network Hub account (secondary)
        → Finds: Transit Gateway route table modified 8 min before symptom
        → Correlates: IAM user in Shared Services account made the change

Step 4: Agent checks Log Archive account (secondary)
        → CloudTrail: "ModifyTransitGatewayVpcAttachment" at 14:52 UTC
        → Confirms: Route to BU-A database VPC attachment was detached

Step 5: Root Cause Identified:
        "Transit Gateway VPC attachment for BU-A database subnet was
         removed during infrastructure maintenance in Network Hub account"

Custom Skills for Landing Zone-Aware Investigation

Teach the agent your organization's account structure:

# Organization Account Structure Skill

Our AWS Organization uses the following structure:

- Network Hub Account (444444444444): Contains all Transit Gateway
  attachments, Route 53 Private Hosted Zones, and AWS Network Firewall.
  All inter-account traffic flows through TGW in this account.

- Log Archive Account (555555555555): Contains organization-wide
  CloudTrail logs, VPC Flow Logs, and AWS Config snapshots from ALL
  accounts. Search here first for cross-account API call correlation.

- BU accounts follow naming: bu-{name}-{env} (e.g., bu-payments-prod)

When investigating connectivity issues:
1. Always check TGW route tables in Network Hub (444444444444)
2. Look for recent route/attachment changes in CloudTrail (555555555555)
3. Check VPC Flow Log rejects in Log Archive for the affected VPC CIDR

When investigating IAM/permission issues:
1. Check CloudTrail in Log Archive for the specific API call
2. Cross-reference with SCPs applied at the OU level
3. Identify if the role was recently modified

Skills for Multi-BU Environments

# Multi-BU Escalation Skill

When an investigation identifies a root cause in a shared/core account
(Network Hub, Log Archive, Shared Services):

1. Document the root cause finding clearly
2. Note the impacting change (who, what, when) from CloudTrail
3. Flag that this affects SHARED infrastructure
4. Recommend that the Platform/SRE team be notified
5. Check if other BU accounts may be affected by the same change
   (assess blast radius using TGW route table scope)

When the root cause is within the BU's own accounts:
1. Standard RCA — deployment change, config drift, resource limits
2. No cross-BU notification needed

Tagging Strategy for Cross-Account Topology Discovery

For the agent to build accurate topology across Landing Zone accounts, use consistent tags:

{
  "devops-agent:app": "checkout-service",
  "devops-agent:bu": "BU-A",
  "devops-agent:env": "production",
  "devops-agent:tier": "frontend|backend|database|network",
  "devops-agent:owner-team": "payments-team",
  "aws:cloudformation:stack-name": "..."
}

The devops-agent: prefix makes it clear these tags support operational intelligence. The agent uses tags to infer relationships between resources even across account boundaries.

Ensuring the Agent Can Find Root Causes Across Accounts

1. Enable Resource Explorer in All Accounts

Resource Explorer is critical for topology discovery of resources not managed by CloudFormation:

# Enable in each secondary account
aws resource-explorer-2 create-index --type AGGREGATOR --region us-east-1

2. Use Consistent Tagging Strategy

The agent uses tags for relationship detection. Apply consistent tags across accounts:

{
  "Application": "checkout-service",
  "Environment": "production",
  "Team": "payments-team",
  "CostCenter": "CC-1234"
}

3. Connect CI/CD Pipelines

For code-to-cloud correlation (the agent tracing a production issue back to a specific commit), connect your pipelines:

• GitHub/GitLab — Add source control integrations in the Agent Space console
• AWS CodePipeline — Automatically detected via CloudFormation

This enables the agent to check: "Was there a deployment in the last N minutes that correlates with symptom onset?"

4. Connect Observability Tools

Beyond AWS-native CloudWatch, connect:

• Datadog — Metrics, logs, traces
• Dynatrace — Behavioral topology mapping
• New Relic — Application performance monitoring
• Splunk — Log aggregation
• Grafana — Dashboards and alerts

5. Configure Alert Sources

Set up investigation triggers so the agent auto-investigates when incidents occur:

• CloudWatch Alarms — Via EventBridge rules
• ServiceNow — Bidirectional integration (creates/updates incidents)
• PagerDuty — Trigger auto-investigations from pages
• Custom webhooks — Via EventBridge or MCP servers

6. Create Custom Skills for Cross-Account Patterns

Upload natural language runbooks that teach the agent your organization's investigation patterns:

# Cross-Account Database Investigation Skill

When investigating database-related latency in the API account:
1. Check RDS metrics in the database account (222222222222)
2. Look for connection pool saturation (max_connections)
3. Check if recent deployments in the CI/CD account modified connection strings
4. Verify VPC peering routes between API and database accounts
5. Check Security Group rules for inter-account communication

---

Understanding the Session Policy (Permission Ceiling)

Critical Architecture Detail

When the agent assumes DevOpsAgentRole in secondary accounts, AWS applies a service-side session policy (AIDevOpsAgentAccessPolicy) as a permissions ceiling:

Effective Permissions = IAM Role Policy ∩ Session Policy (AIDevOpsAgentAccessPolicy)

This means:

• ✅ Allowed: Read-only metadata operations (Describe*, Get*, List*)
• ❌ Blocked: Data-plane read operations (sqs:ReceiveMessage, dynamodb:GetItem)
• ❌ Not bypassed by: Adding inline policies to the role

Recently Added Permissions (June 2026)

• athena:GetQuery*, athena:StartQueryExecution — Enables Security Lake queries
• glue:GetPartitions — For Athena catalog access

Workaround: BYO MCP Servers

For permissions outside the session policy ceiling, build a custom MCP server:

┌─────────────────┐         ┌──────────────────────┐
│  Agent Space     │  HTTP   │  Custom MCP Server    │
│  (DevOps Agent) │───────> │  (Lambda/ECS)         │
│                  │  SigV4  │  Own IAM Role         │
│  Session Policy  │         │  (No session policy)  │
│  APPLIES here   │         │  Full permissions     │
└─────────────────┘         └──────────────────────┘

The MCP server runs under its own execution role — not subject to the DevOps Agent session policy.

---

Monitoring Agent Activity with EventBridge

Since CloudTrail visibility of agent actions in secondary accounts is limited, use EventBridge to track investigation lifecycle:

{
  "source": ["aws.aidevops"],
  "detail-type": ["Investigation Status Change"]
}

Route these events to CloudWatch Logs, S3, or your SIEM for audit compliance.

---

Common Pitfalls & Troubleshooting

Issue	Root Cause	Fix
Agent only investigates one region	SCP with aws:RequestedRegion condition	Remove region restriction for the DevOps Agent role, or add all required regions
"Invalid" status on secondary account	IAM role trust policy mismatch	Re-validate: remove and re-add the account association
Agent doesn't see resources	Resource Explorer not enabled	Enable Resource Explorer aggregator index in the account
Topology shows stale data	Auto-refresh is every ~24 hours	Trigger manual refresh via ValidateAwsAssociations API
Shared account role naming conflict	Multiple Agent Spaces targeting same account	Use stage-specific role names with suffixes
Cross-account Athena queries fail	Glue catalog permissions missing	Add glue:GetPartitions, glue:GetTable to inline policy

---

Enterprise Governance Considerations

IAM Identity Center Integration

When using IAM Identity Center (IDC):

• Users access the Operator Web App via IDC authentication
• The service assumes DevOpsAgentRole-WebappIDC-abc123 on behalf of the user
• IDC identity is propagated for auditing only (CloudTrail onBehalfOf field)
• Restrict Custom Agent creation with deny policies on the WebappIDC role

Restricting Agent Access in Specific Accounts

Use IAM conditions to limit what the agent can investigate:

{
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "StringNotEquals": {
      "aws:RequestedRegion": ["us-east-1", "eu-west-1"]
    }
  }
}

Concurrent Investigation Limits

• 3 concurrent investigations per Agent Space (queued beyond that)
• For high-alert-volume environments, consider multiple Agent Spaces to parallelize

---

Summary: Deployment Decision Framework

Quick Reference: Which Accounts to Add as Secondaries

Your Agent Space Purpose	Secondary Accounts to Add
BU-specific workload	BU prod/staging accounts + Network Hub + Log Archive + CI/CD Tooling
Platform/SRE (central)	Network Hub + Log Archive + Security Tooling + ALL BU prod accounts
Security Operations	Security Tooling + Log Archive + All prod accounts (for security incident RCA)
Single application	App's prod + staging + the app's CI/CD + Network Hub + Log Archive

Multi-BU Decision Matrix

Question	Centralized Model	Federated Model	Hybrid Model
Who manages networking?	Central SRE	Each BU	Central SRE
Who manages logging?	Central SRE	Each BU	Central SRE
Who manages CI/CD?	Central SRE	Each BU	Each BU
Can BU-A agent see BU-B resources?	Only shared infra	No	Only shared infra
Who owns the DevOps Agent account?	Central SRE	Central SRE (creates spaces); BU operates	Central SRE
Agent Space per...?	Per-service (cross-BU)	Per-BU	Per-BU-service
Network Hub added to...?	All Agent Spaces	BU-specific + Platform	All Agent Spaces
Log Archive added to...?	All Agent Spaces	Org-level space only	All Agent Spaces

Landing Zone Alignment Checklist

• [ ] Create dedicated DevOps Agent account under Infrastructure OU
• [ ] Deploy Agent Space(s) in the dedicated account
• [ ] Add Log Archive as secondary to every Agent Space (org-wide CloudTrail)
• [ ] Add Network Hub as secondary (for connectivity RCA)
• [ ] Deploy secondary role via OU-targeted StackSets with auto-deployment
• [ ] Exempt DevOps Agent role from region-deny SCPs
• [ ] Create "Organization Account Structure" custom skill for each Agent Space
• [ ] Use unique role names when shared accounts serve multiple Agent Spaces
• [ ] Enable Resource Explorer in all workload accounts (StackSet deployment)
• [ ] Implement consistent tagging strategy across all accounts

┌─ How many accounts?
│
├── 50–100 accounts
│   └── 5–15 Agent Spaces (per-service/team)
│       └── Deploy via CDK or CloudFormation StackSets
│
├── 100–400 accounts
│   └── 10–40 Agent Spaces
│       └── Consider regional isolation
│       └── Deploy via StackSets + auto-deployment
│
└── 400–1000+ accounts
    └── Open SpecReq for tailored guidance
        └── Consider tiered Agent Space architecture
        └── BYO MCP for custom cross-account tooling

Key principles:

1. Dedicated monitoring account — Separate Agent Space infrastructure from workloads
2. Per-service Agent Spaces — Scoped topology = better accuracy
3. Separate prod from pre-prod — Different Agent Spaces for different lifecycle stages
4. Enable Resource Explorer everywhere — Critical for topology discovery
5. Connect all signal sources — CI/CD, observability, ITSM — more data = better RCA
6. Custom Skills — Encode your organization's investigation knowledge

---

Additional Resources

• AWS DevOps Agent Documentation
• Getting Started with CDK
• Getting Started with CloudFormation
• Connecting Multiple AWS Accounts
• How DevOps Agent Uses Multi-Agent Reasoning
• Best Practices for Deploying in Production
• What is a DevOps Agent Topology?
• 225 Pre-Built Skills (GitHub)
• Interactive Demo (6 break/fix scenarios)

---

This article is authored by Lokesh Gupta, Technical Account Manager at AWS Enterprise Support. The strategies described are based on field experience deploying AWS DevOps Agent with Enterprise customers operating multi-account AWS environments.