ProvisionedThroughputExceededException

Question

Hello,

I am getting below exception when two glue jobs consumes same stream. The Stream has multiple shards.
com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for Shard - 826756392126/dev-vault-stream/shardId-000000000000 (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: dd6d1474-67a6-59c1-8094-f09703c5ef9c; Proxy: null)

My understanding is Kinesis can have a multiple consumers. Could you please help me here to understand the issue and how to fix it?

Thanks

Answer

You can have multiple consumers but they share the same shard throughput of 2MB/s when provisioned.   
You could add more shard to reduce the volume for each one but it's more robust to enable Fan-out reading, where readers don't compete for bandwidth.   
You have to first enable it on the shard (or use on demand mode) and then configure the Glue 4 streaming job to use it:   
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-kinesis-efo.html

ProvisionedThroughputExceededException

Conteúdo relevante