AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
Amazon Bedrock Advanced Operations Playbook: Optimizing Performance, Cost, and Availability
This playbook provides guidance on advanced techniques to optimize your Amazon Bedrock implementation for performance, cost efficiency, and availability.
Introduction
Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies through a unified API. This playbook provides guidance on advanced techniques to optimize your Amazon Bedrock implementation for performance, cost efficiency, and availability.
We'll cover key strategies including prompt caching, cross-region inference, model switching, and regional considerations to help you build robust, efficient, and scalable generative AI applications.
Understanding Amazon Bedrock's Core Optimization Features
Before diving into implementation details, it's essential to understand how Amazon Bedrock's optimization features work together to reduce costs and improve performance.
1. Prompt Engineering Techniques
The way you structure your prompts fundamentally determines your token consumption on Bedrock. Each foundation model uses its own tokenizer and processes text differently, which means the same prompt can result in varying token counts across models. Understanding these differences is crucial for optimization, and Bedrock provides built-in tools to help you monitor and analyze your token usage patterns.
Effective prompt engineering is one of the most impactful ways to reduce both token consumption and latency. By crafting concise, well-structured prompts, you can achieve the same results while using fewer tokens and receiving faster responses.
Prompt Caching takes optimization further by allowing you to cache portions of your context that remain consistent across multiple requests. This technique can reduce inference response latency and input token costs by up to 85%, making it particularly valuable for applications with long system prompts or repeated context, such as chatbots that process the same document across multiple user questions.
Intelligent Prompt Routing adds another layer of optimization by automatically selecting the best foundation model for each specific task. Rather than manually choosing between models, intelligent routing dynamically evaluates your prompt and routes it to the model that offers the optimal balance of response quality and cost. This means simpler queries can be handled by more cost-effective models, while complex reasoning tasks are automatically directed to more capable models.
Together, these techniques form a comprehensive approach to optimization: engineer your prompts efficiently, cache what you can reuse, and let intelligent routing ensure each request uses the most appropriate model for the job.
2. Cross-Region Inference (CRIS)
Cross-region inference enables you to seamlessly manage unplanned traffic bursts by utilizing compute across different AWS Regions while enabling higher throughput.. Amazon Bedrock offers two types of cross-Region inference profiles:
Geographic Cross-Region Inference: Automatically selects the optimal AWS Region within a specific geography (US, EU, APAC, etc.) to process your inference request while maintaining data residency within geographic boundaries.
Global Cross-Region Inference: Automatically selects the optimal AWS Region worldwide to process requests, providing the highest available throughput and approximately 10% cost savings.
3. Inference Profiles
Inference profiles define a model and one or more Regions to which requests can be routed, allowing you to track usage metrics, monitor costs with tags, and implement cross-region inference.
4. Model Selection and Switching
Amazon Bedrock provides access to over 100 foundation models from providers like Anthropic, AI21 Labs, Cohere, Meta, Stability AI, and Amazon's own Nova models, each with different capabilities, pricing, and regional availability. See the full list of foundational models available on Bedrock in our documentation.
5. Optimizing max_tokens for Quota Efficiency
Optimizing the max_tokens parameter is one of the most effective ways to maximize your throughput and quota efficiency in Amazon Bedrock. By setting max_tokens to closely match your expected response sizes, you enable more concurrent requests and better utilize your tokens-per-minute (TPM) quota. Amazon Bedrock initially reserves your specified max_tokens value from your quota, then replenishes unused tokens after the response is generated. This means a well-tuned max_tokens value keeps your quota available for other requests. Certain models like Claude Sonnet 4 and Opus 4 use a 5x burndown rate for output tokens, where each output token consumes five tokens from your throttling quota however you're only billed for actual usage. See the token burndown documentation for the complete list of models and their burndown rates.
Choosing Your Optimization Path
Each optimization technique addresses different needs and can be implemented independently:
- Start with Prompt Caching if you have repetitive context or long system prompts
- Use Cross-Region Inference when you need higher throughput or better availability
- Implement Intelligent Routing to automatically optimize cost vs. quality tradeoffs
- Set up Monitoring to measure the impact of your optimizations
Setting Up Model Invocation Logging
Model invocation logging collects invocation logs, model input data, and model output data for all invocations in your AWS account within a Region. This provides visibility into:
- Full request and response data
- Metadata associated with all model invocations
- Token usage and latency metrics
- Model performance across different use cases
We recommend to enable Model invocation logging. You can log to CloudWatch Logs, Amazon S3, or both:
- CloudWatch Logs: Best for real-time monitoring and alerting. Supports JSON invocation log events up to 100 KB
- Amazon S3: Best for long-term storage and analysis. Supports large payloads and binary data. Data can be queried using Amazon Athena or processed with AWS Glue
For detailed information on setting up model invocation logging, see Monitor model invocation using CloudWatch Logs and Amazon S3.
Implementing Prompt Caching for Performance and Cost Optimization
Prompt caching is the most effective way to reduce latency, tokens per minute utilization and costs when working with repetitive contexts.
When to Use Prompt Caching
Prompt caching is recommended as a default best practice, particularly where you have larger system prompts, but is very effective for:
- Chatbots where users upload documents and ask multiple questions about them
- Applications with long, static system prompts
- Use cases requiring repeated processing of the same context. It can be used to cache and checkpoint "System", "Messages" and "Tools" fields.
Supported Models for Prompt Caching
Prompt caching is available for select models, see Supported models, Regions, and limits
Implementation Example
Here's how to implement prompt caching with the Converse API:
import boto3 import json bedrock_runtime = boto3.client('bedrock-runtime') # Example with system prompt caching response = bedrock_runtime.converse( modelId='global.anthropic.claude-sonnet-4-5-20250929-v1:0', system=[ { "text": "You are an assistant that helps create summaries of financial documents." }, { "cachePoint": { "type": "default" } } ], messages=[ { "role": "user", "content": [ { "text": "Summarize this quarterly report: [document content]" }, { "cachePoint": { "type": "default" } } ] } ] )
Best Practices for Prompt Caching
- Identify Static Content: Cache static portions of your prompts like system instructions, tools definitions, or reference documents.
- Meet Minimum Token Requirements: Ensure your cache checkpoints meet the minimum token requirements for your model (e.g., 1,024 tokens for Claude 3.7 Sonnet).
- Maintain Consistency: Keep cached content consistent between requests to avoid cache misses.
- Monitor Cache Hits: Track cache hit rates to ensure your caching strategy is effective.
- Optimize Context Window Size: Set
max_tokensvalue according to your use case and needs. While models support 200K or 1M tokens, larger contexts significantly increase time-to-first-token (TTFT) and overall latency. Include only the context necessary for your specific task, sending 200K tokens when you need 10K degrades performance without improving quality. Benchmark different context sizes to find the optimal balance for your use case. - Consider TTL: Remember that caches have a 5-minute Time To Live (TTL) that resets with each successful cache hit.
Prompt Caching Monitoring
Monitoring your prompt caching implementation is crucial for optimizing performance and cost. When you use prompt caching, Amazon Bedrock automatically publishes cache-specific metrics CacheReadInputTokens and CacheWriteInputTokens to CloudWatch in the AWS/Bedrock namespace.
The cache hit/miss ratio is a critical metric that indicates the effectiveness of your caching strategy. A high hit ratio means your cache is being utilized efficiently, while a low hit ratio suggests optimization opportunities.
Query Cache Metrics from CloudWatch
Use the CloudWatch API to query cache performance metrics and calculate hit/miss rates:
import boto3 from datetime import datetime, timedelta cloudwatch = boto3.client('cloudwatch') # Query cache metrics for the last 24 hours end_time = datetime.utcnow() start_time = end_time - timedelta(hours=24) response = cloudwatch.get_metric_data( MetricDataQueries=[ { 'Id': 'cache_reads', 'MetricStat': { 'Metric': { 'Namespace': 'AWS/Bedrock', 'MetricName': 'CacheReadInputTokens', 'Dimensions': [ { 'Name': 'ModelId', 'Value': 'anthropic.claude-sonnet-4-20250514-v1:0' } ] }, 'Period': 3600, 'Stat': 'Sum' } }, { 'Id': 'cache_writes', 'MetricStat': { 'Metric': { 'Namespace': 'AWS/Bedrock', 'MetricName': 'CacheWriteInputTokens', 'Dimensions': [ { 'Name': 'ModelId', 'Value': 'anthropic.claude-sonnet-4-20250514-v1:0' } ] }, 'Period': 3600, 'Stat': 'Sum' } }, { 'Id': 'cache_hit_rate', 'Expression': 'cache_reads / (cache_reads + cache_writes) * 100', 'Label': 'Cache Hit Rate (%)' }, { 'Id': 'cache_miss_rate', 'Expression': 'cache_writes / (cache_reads + cache_writes) * 100', 'Label': 'Cache Miss Rate (%)' } ], StartTime=start_time, EndTime=end_time ) # Display results for result in response['MetricDataResults']: if result['Id'] in ['cache_hit_rate', 'cache_miss_rate']: avg_value = sum(result['Values']) / len(result['Values']) if result['Values'] else 0 print(f"{result['Label']}: {avg_value:.2f}%") else: total = sum(result['Values']) print(f"{result['Id']}: {total:,.0f} tokens")
Creating CloudWatch Dashboards for Cache Metrics
Set up a CloudWatch dashboard to visualize your cache performance using metrics:
import boto3 import json def create_cache_monitoring_dashboard(): """Create a CloudWatch dashboard for monitoring cache performance""" cloudwatch = boto3.client('cloudwatch') dashboard_body = { "widgets": [ { "type": "metric", "x": 0, "y": 0, "width": 8, "height": 6, "properties": { "metrics": [ ["AWS/Bedrock", "CacheReadInputTokens", {"stat": "Sum", "label": "Cache Hits"}], [".", "CacheWriteInputTokens", {"stat": "Sum", "label": "Cache Misses"}] ], "view": "pie", "region": "us-east-1", "title": "Cache Hit vs Miss Distribution", "period": 86400 } }, { "type": "metric", "x": 8, "y": 0, "width": 16, "height": 6, "properties": { "metrics": [ ["AWS/Bedrock", "CacheWriteInputTokens", {"stat": "Sum"}], [".", "CacheReadInputTokens", {"stat": "Sum"}] ], "view": "timeSeries", "stacked": False, "region": "us-east-1", "title": "Cache Write vs Read Tokens Over Time", "period": 300, "yAxis": { "left": { "label": "Tokens" } } } }, { "type": "metric", "x": 0, "y": 6, "width": 24, "height": 6, "properties": { "metrics": [ [{"expression": "m1 / (m1 + m2) * 100", "label": "Cache Hit Rate (%)", "id": "e1"}], ["AWS/Bedrock", "CacheReadInputTokens", {"id": "m1", "visible": False, "stat": "Sum"}], [".", "CacheWriteInputTokens", {"id": "m2", "visible": False, "stat": "Sum"}] ], "view": "timeSeries", "stacked": False, "region": "us-east-1", "title": "Cache Hit Rate Percentage", "period": 300, "yAxis": { "left": { "min": 0, "max": 100, "label": "Percentage" } } } } ] } cloudwatch.put_dashboard( DashboardName="BedrockCacheMonitoring", DashboardBody=json.dumps(dashboard_body) )
Analyzing Cache Performance
To optimize your prompt caching implementation, analyze these key dimensions:
- Cache Hit Ratio: Aim for a high cache hit ratio (>80%) for optimal performance and cost savings. Low hit ratio may indicate inconsistent prompts or cache checkpoints not meeting token requirements.
- Cache Write vs Read Patterns: High cache writes indicate new content being cached or cache misses. High cache reads indicate effective cache utilization. Sum these metrics over a defined period to understand your caching efficiency.
- Latency Comparison: Compare latency between cache hits and misses. Significant latency reduction on cache hits confirms caching is working effectively.
- Token Cost Savings: Calculate the cost difference between cached and non-cached tokens. Track the number of tokens read from cache vs. written to cache to estimate savings.
- Cache Expiration Patterns: Monitor cache misses that occur after periods of inactivity. If cache misses frequently occur after the 5-minute TTL, consider implementing a cache warming strategy.
- Cache Checkpoint Distribution: Analyze which cache checkpoints are most frequently hit. This helps identify which parts of your prompts benefit most from caching.
Leveraging Cross-Region Inference for Increased Throughput and Availability
Cross-region inference helps manage traffic spikes and increases overall throughput by distributing requests across multiple AWS Regions. We recommend that you use Global Cross-Region Inference by default and you select Geographic Cross-Region Inference if you have strong requirements. See Bedrock inference profiles documentation for more details.
Using Inference Profiles
To use Global cross-region inference, specify a cross-region inference profile when making API calls:
import boto3 import json bedrock = boto3.client('bedrock-runtime', region_name='us-east-1') model_id = "global.anthropic.claude-sonnet-4-5-20250929-v1:0" # <-- Global CRIS inference profile response = bedrock.converse( messages=[{"role": "user", "content": [{"text": "Explain cloud computing in 2 sentences."}]}], modelId=model_id, ) print("Response:", response['output']['message']['content'][0]['text']) print("Token usage:", response['usage']) print("Total tokens:", response['usage']['totalTokens'])
To use Geographic cross-region inference, specify a cross-region inference profile when making API calls:
import boto3 import json bedrock = boto3.client('bedrock-runtime', region_name='us-east-1') model_id = "us.anthropic.claude-sonnet-4-5-20250929-v1:0" # <-- "us" CRIS inference profile response = bedrock.converse( messages=[{"role": "user", "content": [{"text": "Explain cloud computing in 2 sentences."}]}], modelId=model_id, ) print("Response:", response['output']['message']['content'][0]['text']) print("Token usage:", response['usage']) print("Total tokens:", response['usage']['totalTokens'])
If you want to dive deeper in how CRIS works under the hood, see Getting started with cross-region inference in Amazon Bedrock blog post.
Implementation with Intelligent Prompt Routing
Intelligent prompt routing provides a single serverless endpoint to efficiently route requests between different foundation models within the same model family. It dynamically predicts the response quality of each model for each request and routes to the model with the best combination of response quality and cost.
Intelligent prompt routing is particularly effective when:
- You want to optimize costs without sacrificing response quality
- Your application handles diverse prompt types with varying complexity
- You want to automatically benefit from new model releases
- You need simplified model management across a model family
Key Benefits
- Optimized Response Quality and Cost: Automatically routes prompts to achieve the best response quality at the lowest cost
- Simplified Management: Eliminates the need for complex orchestration logic
- Future-Proof: Automatically incorporates new models as they become available
Using A Prompt Router
import boto3 import json bedrock_runtime = boto3.client('bedrock-runtime') # Using a prompt router (requires ARN) response = bedrock_runtime.converse( modelId='arn:aws:bedrock:us-east-1:123456789012:prompt-router/my-router-id', # Prompt router ARN messages=[ { "role": "user", "content": [ { "text": "Explain cloud computing in simple terms." } ] } ] ) # The response includes which model was selected print(f"Model used: {response['ResponseMetadata']['HTTPHeaders'].get('x-amzn-bedrock-model-id')}")
Model Fallback Strategies for Resilience and Optimization
Different models have varying capabilities, costs, and regional availability. Building resilient applications requires strategies that handle model failures gracefully while optimizing for both performance and cost.
A robust model fallback strategy involves several key practices:
- Graceful Degradation: Implement fallback mechanisms to automatically switch to alternative models during failures or throttling events. This ensures your application remains available even when your primary model is unavailable.
- Cost-Aware Selection: Use less expensive models for simpler tasks and reserve premium models for complex reasoning. This balances quality with cost efficiency across your application.
- Regional Availability: Select models available in your primary operating regions and configure fallbacks for models with broader regional coverage to handle regional outages.
- Continuous Evaluation: Use A/B testing to evaluate different models for your specific use cases, and track performance metrics to inform ongoing model selection decisions.
The following implementation demonstrates a practical fallback pattern that retries with a primary model before falling back to a backup model.
Implementation Example for Model Switching
import boto3 import time bedrock_runtime = boto3.client('bedrock-runtime') def invoke_with_fallback(primary_model, backup_model, prompt, max_retries=3): """Invoke a model with fallback to another model if needed""" for attempt in range(max_retries): try: # Try primary model first response = bedrock_runtime.converse( modelId=primary_model, messages=[ { "role": "user", "content": [{"text": prompt}] } ] ) return response except Exception as e: print(f"Error with primary model: {e}") if attempt == max_retries - 1: # Try backup model as last resort try: response = bedrock_runtime.converse( modelId=backup_model, messages=[ { "role": "user", "content": [{"text": prompt}] } ] ) return response except Exception as backup_error: raise Exception(f"Both primary and backup models failed: {backup_error}") time.sleep(1) # Wait before retry
Monitoring and Optimization
To ensure optimal performance and cost efficiency, implement comprehensive monitoring using Amazon Bedrock's built-in capabilities.
Key Metrics to Monitor
- Latency
- Track response times across models and regions
- Compare latency between cache hits and misses
- Monitor p50, p90, and p99 percentiles for consistent performance
- Token Usage
- Monitor input and output token consumption
- Track tokens read from cache vs. written to cache
- Identify opportunities for prompt optimization
- Error Rates
- Track failures by model, region, and error type
- Monitor throttling events and quota utilization
- Set up CloudWatch alarms for critical error thresholds
- Cost
- Monitor costs by model, region, and application
- Use inference profile tags for granular cost allocation
Usage Metrics with Inference Profiles
Application inference profiles enable detailed usage tracking and cost monitoring by using Tagged Inference Profiles:
import boto3 bedrock = boto3.client('bedrock') # Create an application inference profile with tags response = bedrock.create_inference_profile( inferenceProfileName='production-claude-sonnet', modelSource={ 'copyFrom': 'arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-sonnet-4-5-20250929-v1:0' }, tags=[ {'key': 'Environment', 'value': 'Production'}, {'key': 'Application', 'value': 'CustomerService'}, {'key': 'CostCenter', 'value': 'Engineering'} ] ) inference_profile_arn = response['inferenceProfileArn']
Token Estimation with CountTokens API
Token counting is model-specific because different models use different tokenization strategies.
The CountTokens API helps you estimate token usage before sending inference requests, enabling cost prediction and prompt optimization without incurring charges. Beyond inference pre-flight checks, token counting is essential during application architecture and design: use it to validate your assumptions about typical request sizes, test different prompt strategies, and establish realistic cost models before deploying to production.
The count returned by CountTokens matches exactly what would be charged if you sent the same input to the model for inference. This accuracy makes it reliable for architectural planning, cost estimation, quota management, and ensuring your prompts fit within model token limits.
The CountTokens API is currently supported for Anthropic Claude models including Claude 3.5 Haiku, Sonnet (v1 and v2), Claude 3.7 Sonnet, and Claude Opus 4 and Sonnet 4. Check our documentation for the latest model and region availability.
Implementation with Converse API
The following example shows how to count tokens before running inference with Claude Sonnet 4. Use this pattern during development to understand your application's token consumption patterns, and in production to make runtime decisions about request routing and cost control:
import boto3 bedrock_runtime = boto3.client("bedrock-runtime") # Prepare your conversation input input_to_count = { "messages": [ { "role": "user", "content": [{"text": "Analyze this document..."}] } ], "system": [{"text": "You are a document analyst."}] } # Count tokens before inference response = bedrock_runtime.count_tokens( modelId="anthropic.claude-sonnet-4-20250514-v1:0", input={"converse": input_to_count} ) token_count = response["inputTokens"] print(f"Estimated tokens: {token_count}") # Decide whether to proceed based on cost/quota if token_count < 10000: # Proceed with inference inference_response = bedrock_runtime.converse( modelId="anthropic.claude-sonnet-4-20250514-v1:0", **input_to_count )
TimeToFirstToken
TimeToFirstToken measures the latency from request submission to the first token received. It applies to streaming APIs (ConverseStream and InvokeModelWithResponseStream) and enables you to establish SLA baselines and detect latency degradation without any client-side instrumentation. Available automatically across all commercial regions with no opt-in required. Metrics update every minute for successfully completed requests.
This metric is particularly useful alongside prompt caching: a sudden increase in TTFT can indicate cache misses causing full context reprocessing, which may point to issues with cache consistency or checkpoint placement.
import boto3 cloudwatch = boto3.client('cloudwatch') # Alarm when p99 TTFT exceeds 2 seconds cloudwatch.put_metric_alarm( AlarmName='Bedrock-HighTTFT-p99', Namespace='AWS/Bedrock', MetricName='TimeToFirstToken', Dimensions=[ {'Name': 'ModelId', 'Value': 'anthropic.claude-sonnet-4-20250514-v1:0'} ], ExtendedStatistic='p99', Period=300, EvaluationPeriods=3, Threshold=2000, # milliseconds ComparisonOperator='GreaterThanThreshold', AlarmDescription='p99 time-to-first-token exceeded 2s over 15 minutes', TreatMissingData='notBreaching' )
EstimatedTPMQuotaUsage
EstimatedTPMQuotaUsage tracks your estimated Tokens Per Minute quota consumption in real time. It accounts for cache write tokens and output token burndown multipliers (e.g., the 5x rate for Claude Sonnet 4 and Opus 4), giving you an accurate picture of quota pressure before throttling occurs. Available across all commercial regions for both cross-region inference profiles and in-region inference.
This pairs directly with the max_tokens optimization strategy in Section 5: by monitoring EstimatedTPMQuotaUsage, you can validate that right-sizing your max_tokens is freeing up quota for additional concurrent requests and set proactive alarms before reaching your quota limit.
import boto3 cloudwatch = boto3.client('cloudwatch') # Alarm when TPM quota usage exceeds 80% cloudwatch.put_metric_alarm( AlarmName='Bedrock-HighTPMQuotaUsage', Namespace='AWS/Bedrock', MetricName='EstimatedTPMQuotaUsage', Dimensions=[ {'Name': 'ModelId', 'Value': 'anthropic.claude-sonnet-4-20250514-v1:0'} ], Statistic='Maximum', Period=60, EvaluationPeriods=5, Threshold=80, # percentage ComparisonOperator='GreaterThanThreshold', AlarmDescription='TPM quota usage exceeded 80%, consider request throttling or a quota increase', TreatMissingData='notBreaching' )
Conclusion
This playbook covered Amazon Bedrock's key optimization techniques: prompt caching for cost reduction, cross-region inference for throughput, intelligent prompt routing for quality-cost balance, and model fallback strategies for resilience. Each technique can be implemented independently based on your immediate needs.
Start with monitoring to understand your baseline, then apply the optimizations that address your specific challenges. As your application evolves, revisit these strategies to ensure you're getting the most value from Amazon Bedrock.
Additional Resources
- Language
- English
Relevant content
asked 10 months ago
- AWS OFFICIALUpdated a year ago
