read_timeout bedrock client, possibly due to large prompts

Question

Running into the following error with bedrock and claude-v2.

for some of the prompts that we are trying, there are intermittent read timeouts with Claude v2. trying to set read_timeout to a higher value did not help, as the timeout happens way before the defined setting. this also started happening when the prompt sizes got bigger (e.g. > 2500 tokens)

**Read timeout on endpoint URL: "https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-v2/invoke"
at 645.8567 seconds
**

Let me know if there are other ways to resolve the timeout issue.

```
boto3_bedrock = bedrock.get_bedrock_client(assumed_role=os.environ.get("BEDROCK_ASSUME_ROLE", None), region=os.environ.get("AWS_DEFAULT_REGION", None))

boto3_bedrock.meta.config.read_timeout = 1500

llm = Bedrock(model_id="anthropic.claude-v2", client=boto3_bedrock, model_kwargs={'max_tokens_to_sample':3072, "temperature": 0.2, "top_p": 0.9, "top_k": 25})

bedrock_embeddings = BedrockEmbeddings(model_id="amazon.titan-embed-text-v1", client=boto3_bedrock)

bedrock_embeddings.client.meta.config.read_timeout = 1500
```

Answer

You can set the timeout when you initialise the bedrock client [[1]](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime.html). Below is a sample of the code to set it up:

```
from boto3 import client
from botocore.config import Config

config = Config(read_timeout=1500)

client = boto3.client(service_name='bedrock-runtime', 
                      region_name='us-east-1',
                      config=config)
```
It looks like you are using a helper method, so you may need to make the change to add the `read_timeout` to the `config` within that.

If you are still getting timeouts you could try `invoke_model_with_response_stream` [[2]](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/invoke_model_with_response_stream.html) as it will start to bring back the response as it generates rather than waiting for the entire processing.

---
[1] https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime.html

[2] https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/invoke_model_with_response_stream.html

read_timeout bedrock client, possibly due to large prompts

Relevanter Inhalt