We have been using Anthropic Claude 3.0 Sonnet model for our enterpirse use. The total input and output tokens which are being processed per minute is ~50-60k which is easily within the limit of 200k defined for claude models. Inspite of this lately we have been receiveing the below error message frequently.
botocore.exceptions.EventStreamError: An error occurred (throttlingException) when calling the InvokeModelWithResponseStream operation: Too many requests, please wait before trying again. You have sent too many requests. Wait before trying again.
Can we get to know as to why this error might be coming? Is this because the 2M token capacity is not just limited to our enterprise but also across all the claude users in that region. And what's the best way to resolve this issue. Is there any way we can increase rate limit or get a dedicated computing resource for our enterprise
Thanks for the response. The provisioned throughput option seems to be costly for small and medium enterprises. AWS should look into increasing the rate limits of the on-demand model or offering a better upgrade plan in Bedrock