By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

ProvisionedThroughputExceededException

0

Hello,

I am getting below exception when two glue jobs consumes same stream. The Stream has multiple shards. com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for Shard - 826756392126/dev-vault-stream/shardId-000000000000 (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: dd6d1474-67a6-59c1-8094-f09703c5ef9c; Proxy: null)

My understanding is Kinesis can have a multiple consumers. Could you please help me here to understand the issue and how to fix it?

Thanks

Topics
AnalyticsInternet of Things (IoT)
Tags
AWS GlueAmazon Kinesis Data StreamsAmazon Kinesis
Language
English
Asmita
asked 2 months ago771 views
1 Answer
1

You can have multiple consumers but they share the same shard throughput of 2MB/s when provisioned.
You could add more shard to reduce the volume for each one but it's more robust to enable Fan-out reading, where readers don't compete for bandwidth.
You have to first enable it on the shard (or use on demand mode) and then configure the Glue 4 streaming job to use it:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-kinesis-efo.html

profile pictureAWS
EXPERT
Gonzalo Herreros
answered 2 months ago
  • Asmita
    2 months ago

    are there any drawbacks of implementing this approach?

  • Gonzalo Herreros EXPERT
    2 months ago

    Obviously has a higher cost on the Kinesis side and it's a relatively new feature on the Glue side, so it's not as well tested as working on the default mode

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content