I am getting below exception when two glue jobs consumes same stream. The Stream has multiple shards. Rate exceeded for Shard - 826756392126/dev-vault-stream/shardId-000000000000 (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: dd6d1474-67a6-59c1-8094-f09703c5ef9c; Proxy: null)

My understanding is Kinesis can have a multiple consumers. Could you please help me here to understand the issue and how to fix it?


asked 2 months ago659 views
1 Answer

You can have multiple consumers but they share the same shard throughput of 2MB/s when provisioned.
You could add more shard to reduce the volume for each one but it's more robust to enable Fan-out reading, where readers don't compete for bandwidth.
You have to first enable it on the shard (or use on demand mode) and then configure the Glue 4 streaming job to use it:

profile pictureAWS
answered 2 months ago
  • are there any drawbacks of implementing this approach?

  • Obviously has a higher cost on the Kinesis side and it's a relatively new feature on the Glue side, so it's not as well tested as working on the default mode

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions