How to remove default partition (__dt='yyyy-mm-dd 00:00:00') created by IoT Analytics datastore mapped to s3 ?

0

Hi Community, Looking for some help with IoT Analytics datastore(s3) partitions. Currently my pipeline routes all the messages to datastore on s3 (customer managed storage) with no custom partitions set.

Here is how they look on s3:

s3://my-bucket/datastore/mydataset/
        __dt=2023-05-23 00:00:00/		
        __dt=2023-05-24 00:00:00/

When I use custom partitions they go under the root partition __dt:

s3://my-bucket/datastore/mydataset/__dt=2023-05-23 00:00:00/
      __partition_device_id=j3e2f83a9A32l_mn/	
      __partition_device_id=JT01WrHb1bthC0B8/
      __partition_device_id=NH6cRS38oPWJQh0k/

The partition by __dt is auto-generated. But I do not want to have this partition for two reasons:

  1. The timestamp is not from my iot event data
  2. Athena has a known issue reading from partitions that have colon(:) in the partition name (this is as per the official docs).

Looking for help if there is a way to remove this default partition and then create my own custom partition as root partition

1回答
0

Here is information about Custom Partition in AWS IOT Analytics:

https://docs.aws.amazon.com/iotanalytics/latest/userguide/custom-partitioning.html

AWS
vtjean
回答済み 1年前
  • Hi @vtjean, thanks for the answer, but my question is not about custom partitions (I already applied them and you can see the example code in my question). I want to disable the default '__dt' partition applied by AWS automatically.

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ