S3 Metadata Query across date folder or hundred thousand folders

Hello,

The decision between using date-wise folders or device-wise folders with date metadata for your IoT data storage in S3 depends on several factors, including your data access patterns, data volume, and performance requirements.

With date-wise folders, your daily batch job can easily locate and process the data for a specific date by looking into the corresponding folder. This simplifies the job's logic and makes it more efficient compared to querying metadata across hundreds of thousands of folders/objects.

Keep in mind that this approach may not be optimal if you frequently need to access data based on device IDs or if you have other use cases that require device-specific data partitioning. Additionally, ensure that you follow best practices for S3 folder structure and object naming conventions to optimize performance and manageability. Also, consider using AWS services like AWS Glue or AWS Athena, which can simplify data querying and processing by automatically partitioning your data based on folder structure or metadata.