Kinesis streams for low-latency apps

Question

A customer is interested in re-factoring their credit-card processing app, and they want to use "streams" to glue the various parts together.  They are considering either Kinesis or Kafka. Besides the fact that Kinesis is not PCI compliance (client side encryption), they are concerned about the 5 read TPS limit per shard.  That seems somewhat low for a real-time app.

The customer only has 100-200 ms time budget to process the transaction.  While I know that Kinesis can pull multiple records per read, this implies a potential 200ms gap between reads.  While not a lot, this may be a show-stopper for my customer.  Is there any way to make Kinesis more responsive (higher read TPS), or is this gap a non-issue in terms of "real-world" latency?!

Sherin_Chandy · Accepted Answer

Even though the throughput limits are not modifiable, customer can achieve higher throughput by sizing their Kinesis streams with enough number of shards and choose partition key wisely to distribute incoming records evenly across the shards.

There is a calculator available on Kinesis console, which can be used to calculate the number of shards needed based on the record rate and size.

Customer also should tune their Kinesis producer and consumer code to optimize the performance using KCL/KPL

Kinesis streams for low-latency apps

Relevant content