How do I improve Amazon Bedrock performance and response times when I process and retrieve large scale data?

Resolution

You might experience latency issues for one of the following reasons:

• Distance between your application and the Amazon Bedrock endpoint
• Larger models that typically require more time to process
• Length and complexity of your prompts
• Large volume of simultaneous API calls
• Shared on-demand capacity pool contention in your AWS Region

To Improve performance and response times, take the following actions.

Choose the right model

Review your specific requirements, and then choose the model that best fits your needs for speed and output quality.

Improve your input and system prompts

Reduce the number of tokens in both your input prompts and system prompts. If your model has less tokens to process and generate, then the model generates a faster response.

It's a best practice to use clear and concise prompts, structured templates, and prompt engineering techniques.

Use prompt caching

To reduce inference response latency and input token costs, use prompt caching. Use implicit prompt caching for an automatic solution that requires no configuration. For more control, use explicit prompt caching and set your cache checkpoints. Place static content, such as system prompts, at the beginning of your prompt, and dynamic content at the end to increase cache usage. For certain models, you can also configure a time to live (TTL) up to 1 hour for longer-running sessions.

Use inference parameters

Use inference parameters to control and tune response generation. Reduce maxTokens to limit output length, as fewer output tokens require less computation. Use temperature, Top K, and Top P to control randomness and diversity. Use stop sequences to end generation early when a specific pattern is matched.

Use latency-optimized inference

Latency-optimized inference for foundation models in Amazon Bedrock provides faster response times and improved responsiveness for AI applications. Set the performanceConfig latency parameter to Optimized when you call the Amazon Bedrock runtime API. Latency-optimized inference is available for a subset of models and Regions.

Use smaller models

Larger models typically have better output quality but higher latency. Consider smaller models that offer faster responses with reduced capabilities. You can also use intelligent prompt routing to automatically route simpler queries to faster, cheaper models within the same model family.

Select a closer AWS Region

If the model isn't available in your AWS Region, then choose a Region in Amazon Bedrock closest to you.

Use Cross-Region Inference

Cross-region inference routes your requests across multiple AWS Regions using inference profiles. You can choose geographic profiles or global profiles. Cross-region inference reduces throttling when you reach regional capacity limits. Cross-region inference improves availability but can add cross-region latency. All data stays on the AWS network and is encrypted in transit.

Use Provisioned Throughput

For production workloads that require consistent, predictable throughput, purchase provisioned throughput to get a dedicated level of capacity for a model at a fixed cost. Provisioned throughput is measured in Model Units (MUs), which specify a rate of input and output tokens per minute. This avoids contention with other customers on the shared on-demand capacity pool.

Use Batch Inference for asynchronous workloads

If your large-scale data processing doesn't require real-time responses, then use batch inference. You submit multiple prompts asynchronously to Amazon Simple Storage Service (Amazon S3), and retrieve output files when the job completes. Batch inference has no timeout constraints and processes large datasets efficiently. Batch inference doesn't support provisioned models or tool calls.

Use service tiers

To optimize for different combinations of availability, cost, and performance, use service tiers. For example, use the Priority tier for fast response times on customer-facing workloads, or Flex for cost savings on latency-tolerant workloads. Set the serviceTier parameter to reserved, priority, default, or flex when you call the Amazon Bedrock runtime API.

Use streaming APIs

The InvokeModel and Converse APIs wait until all response tokens are generated before they send the tokens back to you. Instead, use the InvokeModelWithResponseStream and ConverseStream APIs, because these APIs don't wait until all tokens are generated. They return the response in a stream.