- 新しい順
- 投票が多い順
- コメントが多い順
Hello,
As you have correctly pointed out, Athena query can only pick up the underlying date partitions and data when the date partition format in 'storage.location.template' matches the actual path in the S3 bucket. Because the '${dt}' path format in 'storage.location.template' depends on the date format defined in 'projection.dt.format', it is not possible to define or map the zero padded format to non zero padded format with Athena Partition Projection.
A possible workaround is using conventional Hive partitions instead of Partition Projection, for example, after created the table without Partition Projection, you can use ALTER TABLE ADD PARTITION query to add and map date partitions to the s3 path manually or programmatically:
ALTER TABLE my_table_name ADD PARTITION (dt = '2022/09/06') LOCATION 's3://mystorage/path/to/2022/9/6/';
To add partitions in batch programmatically, you can construct the Athena queries using a custom python script and execute them via start_query_execution API. Reference: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/athena.html#Athena.Client.start_query_execution
Thanks very much for the answer. I was afraid of that. We switched from the old "ADD PARTITION" method to "PARTITION PROJECTION" so as to make things easier, but in the case of leading zeros, it didn't. :(
関連するコンテンツ
- AWS公式更新しました 3年前