Kinesis Stream Best Practices and Calculations

0

A customer is looking at increasing consumers for our Kinesis Stream.

Currently they have a consumer script that polls every min, which they want to add firehose to consume as well.

What are the constraints and limits that they should be looking out for, and how to calculate the optimal number of shards required to support the increased number of consumers?

Is this approach even correct?

AWS
asked 4 years ago2541 views
1 Answer
1
Accepted Answer

I came across this Kinesis Shard Calculator which takes inputs as number of Consumers, Message Size, Throughput, Consumption speed and then recommends the number of Shards. Try it !!

https://comcastsamples.github.io/KinesisShardCalculator/

Amazon Kinesis Data Streams has the following stream and shard quotas and limits.

  • There is no upper quota on the number of streams you can have in an account.

  • The default shard quota is 500 shards per data stream for the following AWS regions: US East (N. Virginia), US West (Oregon), and Europe (Ireland). For all other regions, the default shard quota is 200 shards per data stream.

  • To request the shards per data stream quota increase, follow the procedure outlined in Requesting a Quota Increase.

  • A single shard can ingest up to 1 MB of data per second (including partition keys) or 1,000 records per second for writes. Similarly, if you scale your stream to 5,000 shards, the stream can ingest up to 5 GB per second or 5 million records per second. If you need more ingest capacity, you can easily scale up the number of shards in the stream using the AWS Management Console or the UpdateShardCount API.

  • The maximum size of the data payload of a record before base64-encoding is up to 1 MB.

  • GetRecords can retrieve up to 10 MB of data per call from a single shard, and up to 10,000 records per call. Each call to GetRecords is counted as one read transaction.

  • Each shard can support up to five read transactions per second. Each read transaction can provide up to 10,000 records with an upper quota of 10 MB per transaction.

  • Each shard can support up to a maximum total data read rate of 2 MB per second via GetRecords. If a call to GetRecords returns 10 MB, subsequent calls made within the next 5 seconds throw an exception.

profile pictureAWS
answered 4 years ago
profile picture
EXPERT
reviewed 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions