Latency of upload a file and download it from s3

For S3, if your comparison is ElastiCache, you will want to consider S3 Express One Zone, which provides the lowest latency option in S3. In discussing latency, you want to think about transmission time and server time.

Server Time can be tracked by FirstByteLatency. The best way to track full round trip latency will be through client side metrics. S3 provides a metric TotalRequestLatency that tracks this from the server side, but realize some aspects of this aren't tracked, like establishing a TCP connection, any client side queuing, etc.

Transmission time for 256KB will be a low number, but the exact details will depend upon your networking, but a reasonable expectation might be single digit ms. In S3 Standard a single connection can deliver 85-90MBs which is about 2.5ms for 256KB.

For the overall requests, with S3 Standard you should budget more than 10ms, with S3 Express One Zone you may be able to budget less than 10ms per request. S3 Express One Zone provides consistent single digit FirstByteLatency, but for your application, you'll need to consider that transmission time, any network latency, and any effect your own code introduces. I'd suggest testing to validate all aspects of your design, but those are some rough starting points. Another advantage of S3 Express One Zone is that latency is also more predictable, as in tail latencies are lower.

If you're looking to dive deeper here I'd suggest this presentation from re:Invent 2023, Deep dive on Amazon S3 Express One Zone storage class (STG230).