Analyze Amazon Bedrock cost spikes with AWS DevOps Agent

Introduction

Organizations that run generative AI workloads on Amazon Bedrock pay by the token. Unlike traditional compute resources where costs scale predictably with instance hours, with Amazon Bedrock one misconfigured application can use millions of tokens in minutes. Retry loops that don't back off, prompts that pass entire documents without truncation, and batch jobs without concurrency controls sometimes generate significant unexpected charges.

The challenge is timing. AWS Cost Explorer updates hours after consumption occurs. By the time a cost anomaly appears in billing dashboards, the damage is done.

However, Amazon Bedrock publishes InputTokenCount and OutputTokenCount metrics to CloudWatch within minutes of every model invocation. These metrics represent the earliest available cost signal in your environment. For more information about token counts, see How tokens are counted in Amazon Bedrock.

For organizations that work with AWS Enterprise Support, the Technical Account Manager (TAM) can help you design monitoring strategies that catch these anomalies early. AWS Cloud Support Engineers regularly assist customers to configure CloudWatch alarms and automate investigation workflows for their generative AI workloads.

This article outlines how to turn CloudWatch token metrics into an automated detection and investigation pipeline that moves from token spike to autonomous root cause analysis in minutes. The solution uses the following AWS services:

• CloudWatch alarms

• Amazon Simple Notification Service (Amazon SNS)

• AWS Lambda

• AWS DevOps Agent

Challenges in detecting Amazon Bedrock cost anomalies

To troubleshoot unexpected spikes in Amazon Bedrock token consumption, you must manually correlate CloudWatch metrics across multiple models and review application logs for invocation patterns. Then, you must use AWS CloudTrail to check recent deployments for specific applications or teams that are causing the spikes. This triage process takes hours.

The following scenarios cause spikes:

• Applications send entire documents as prompts without chunking or truncating them.

• SDK retry logic amplifies transient errors into hundreds of duplicate invocations.

• Batch processing jobs run without concurrency controls.

• Teams test developments against production models without rate limitations.

AWS DevOps Agent is a frontier agent that autonomously investigates incidents and correlates metrics, logs, and application topology. It delivers root cause analysis and remediation steps in minutes. By using webhooks to connect CloudWatch alarms to DevOps Agent, the investigation automatically starts so that you don't need to manually correlate logs.

Customer use case

Consider the following example use case.

A financial services company runs multiple generative AI workloads on Amazon Bedrock. Their applications include a customer-facing chatbot, an internal document summarization service, and a code review assistant. Each application calls different foundation models.

One morning, a developer pushes a code change to the summarization service that inadvertently removes a token quota check. The service begins sending entire PDF documents that exceed 100 pages in length as single prompts to the model. Within 30 minutes, the service consumes 2 million input tokens, and generates significant unexpected charges.

Under normal circumstances, the team doesn't discover the issue until the next billing cycle review. However, with the automated cost spike detection pipeline that this article outlines, the CloudWatch alarm activates within 5 minutes of the spike. AWS DevOps Agent autonomously investigates, identifies the summarization service as the source, and uses CloudTrail to correlate the spike with the recent deployment. Then, the agent recommends rolling back the change and implementing input validation.

This pattern applies to organizations that run Amazon Bedrock workloads at scale, whether processing customer support tickets, generating marketing content, or building AI-powered search. The common thread is that token consumption can spike unexpectedly, and the cost signal arrives too late through traditional billing channels.

Solution overview

This solution uses CloudWatch, Amazon SNS, and Lambda to route Amazon Bedrock token consumption alerts to DevOps Agent for automated investigation.

The automated cost spike detection pipeline flow includes the following steps:

1. Amazon Bedrock automatically publishes InputTokenCount to CloudWatch, without the use of tools.

2. A CloudWatch alarm continually evaluates the metric and transitions to the ALARM state when the sum exceeds the threshold in a 5-minute window.

3. The alarm publishes a notification to an Amazon SNS topic.

4. Amazon SNS invokes a Lambda function.

5. The Lambda function builds a signed webhook payload with Amazon Bedrock specific investigation context and sends it to DevOps Agent.

6. DevOps Agent autonomously investigates the spike.

Solution implementation

Prerequisites

Before you create an automated cost spike detection pipeline, you must take the following actions:

• Configure an AWS DevOps Agent Space with access to the AWS account that contains your Amazon Bedrock resources. For instructions, see Creating an Agent Space.

• Confirm that you can access the Amazon Bedrock model, and verify that active workloads are generating metrics.

• Configure a DevOps Agent webhook with Hash-based Message Authentication Code (HMAC) credentials that you store in AWS Secrets Manager.

• Create an Amazon SNS topic to receive alarm notifications.

• Create a Lambda execution role that has permissions to read from Secrets Manager and allows Amazon SNS to invoke the role.

• Grant DevOps Agent the required AWS Identity and Access Management (IAM) permissions to access CloudWatch and Amazon Bedrock resources, following the principle of least privilege.

Technical scope and limitations

DevOps Agent is available in specific AWS Regions. The agent monitors resources from a supported Agent Space Region in a Region that you choose.

The agent works with AWS service telemetry and can't view application-level code or business logic. Deeper application-specific troubleshooting might still require manual investigation.

Configure CloudWatch alarms for Amazon Bedrock

CloudWatch alarms for Amazon Bedrock token metrics provide operational visibility.

The following alarm monitors InputTokenCount in the Amazon Bedrock namespace:

AlarmName: bedrock-input-token-spike
Namespace: AWS/Bedrock
MetricName: InputTokenCount
Statistic: Sum
Period: 300
EvaluationPeriods: 1
DatapointsToAlarm: 1
Threshold: 50000
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
AlarmActions: arn:aws:sns:us-east-1:EXAMPLE_ACCOUNT_ID:bedrock-cost-alerts

If Amazon Bedrock consumes more than 50,000 input tokens in one 5-minute period, then the alarm triggers. To test the alarm, update the input tokens Threshold value to 10,000 and then check that the alarm triggers.

To scope the alarm to a specific model, add the following Dimensions entry:

Dimensions:
  - Name: ModelId
    Value: anthropic.claude-sonnet-4-6

Threshold guidance

Monitor the baseline usage for 1–2 weeks. Set the production threshold at 3–5 times the observed peak. For variable workloads, use CloudWatch anomaly detection for dynamic baselines.

Set up the SNS topic and Lambda function

Create an Amazon SNS topic that receives alarm notifications and subscribes a Lambda function to the alarm. The Lambda function receives the SNS message that contains the alarm details and reads webhook credentials from Secrets Manager. Then, the function builds a payload with an Amazon Bedrock specific investigation context that's similar to the following example:

"Amazon Bedrock input token consumption spiked. Investigate: which model was invoked, invocation frequency, prompt sizes, recent deployments, and whether retry logic or batch processing is amplifying calls."

The function signs the payload with HMAC-SHA256 and sends it to the DevOps Agent webhook endpoint.

After you create the Amazon SNS topic, configure an AWS DevOps Agent Space with access to the account that contains your Amazon Bedrock resources.

Beyond automated detection: Other ways to investigate

This article focuses on automated webhook-triggered investigations, but you can also use the following DevOps Agent interfaces to investigate Amazon Bedrock cost spikes:

• On-demand through Chat:
  Use DevOps Agent Chat  to query your environment in natural language. This interface is useful when you must interactively explore cost anomalies in the CloudWatch dashboard. You can use natural language queries with Chat, such as: "What caused the InputTokenCount spike on Claude Sonnet between 14:00 and 14:30 UTC today?”

  

• Manual investigation:
  Start an investigation directly from the DevOps Agent web app with custom context. Use this interface to analyze spikes that occurred in the past that didn't trigger an alarm.

Each of the preceding methods feeds into the same investigation engine. The automated approach in this article investigates spikes for you. Use the chat and manual options to flexibly investigate specific spikes and tune alarm thresholds.

How DevOps Agent investigates Amazon Bedrock cost spikes

When the pipeline triggers DevOps Agent, it completes the following steps:

1. Detects the anomaly and identifies the spike level and the metric that exceeded the threshold.

2. Correlates source activity, analyzes Amazon Bedrock invocation patterns, identifies which models received elevated traffic, and quantifies token consumption.

3. Checks for contributing factors, and then reviews CloudTrail for recent deployments, configuration changes, or new IAM activity that correlates with the spike timing.

4. Rules out infrastructure issues, and confirms whether throttling, errors, or latency contributed to the spike.

5. Delivers root causes and mitigation and provides specific remediation steps.

In the previous example use case, DevOps Agent identifies the summarization service as the source and traces the spike to the deployment that removed the token quota. Then, it quantifies that the impact is 2 million tokens in 30 minutes. Finally, Dev Ops Agent recommends rolling back the change and adding Amazon Bedrock Guardrails for token budget enforcement.

Proactive prevention

Beyond reactive investigation, DevOps Agent analyzes patterns across incidents to provide proactive recommendations.

Recommendations for Amazon Bedrock cost monitoring include the following categories:

• Observability improvements:
  For example, add alarms for OutputTokenCount and invocations metrics for each model to detect cost anomalies across different dimensions.

• Application hardening:
  For example, implement input validation, token budgets through Amazon Bedrock Guardrails, and rate limiting at the application layer.

In the Prevention tab of the DevOps Agent web app, view your recommendations and provide feedback to refine future suggestions.

Cost considerations

For AWS DevOps Agent, you pay only for the time that the agent spends on operational tasks. For information about prices, see AWS DevOps Agent pricing. The supporting services, CloudWatch alarms, SNS notifications, and Lambda invocations, incur standard AWS charges. To avoid unnecessary costs, delete unused resources after you test.

Conclusion

This article outlines how to use CloudWatch metrics to detect Amazon Bedrock token consumption spikes and automate root cause investigation with AWS DevOps Agent. The pipeline detects anomalies within minutes of occurrence and delivers actionable findings without manual log correlation.

To get started, create a DevOps Agent Space, configure a webhook, and deploy the CloudWatch alarm and Lambda function. After you complete the basic setup, extend the pipeline by adding alarms for OutputTokenCount, configuring thresholds for each model, and integrating with Slack or ServiceNow for incident coordination.

Note: Amazon Bedrock publishes InputTokenCount and OutputTokenCount metrics to CloudWatch within minutes of invoking the model. The token count metrics are your environment’s earliest cost signal. For more information about token counts, see How tokens are counted in Amazon Bedrock. 

About the Author

Santosh Jade is a Technical Account Manager at AWS who helps U.S. federal customers optimize cloud operations and accelerate innovation through Generative AI. He helps agencies to unlock mission outcomes with GenAI-powered solutions for intelligent automation, modernization, and operational efficiency. Outside work, he enjoys photography, cricket, and exploring new destinations and cuisines.

Siddhesh Chavan is an Enterprise Support Technical Account Manager that specializes in the financial services industry at AWS in the Northern Virginia. He specializes in storage, AWS Backup, AWs DevOps Agent and containers. Siddhesh helps customers use generative AI to simplify their use cases and accelerate business outcomes. He’s passionate about delivering reliable, secure solutions at scale.