Does Kinesis Firehose destination support Glue table format of type Iceberg

0

Kinesis Firehose allows to configure S3 as a destination and in the Parquet section allows selecting a Glue catalog table of format Iceberg. However I had little to no luck querying the data.

Does Kinesis Firehose support Glue Iceberg as destination ? Any plans in near future if not.

RegeR
已提問 24 天前檢視次數 84 次
1 個回答
1
已接受的答案

Currently, Kinesis Data Firehose does not natively support direct streaming to AWS Glue with Apache Iceberg table formats as a destination. Kinesis Data Firehose can output to Amazon S3 in formats such as Parquet, which can then be used in conjunction with AWS Glue for further processing and management in Iceberg format. Once the data is in S3, you can utilize AWS Glue to manage and catalog these data into Iceberg format, which supports complex and analytical workloads with robust data management features.

For your specific situation where you had difficulties querying data possibly streamed via Firehose and managed as Iceberg tables in AWS Glue, you might need to check the setup or integration process between S3, AWS Glue, and the Iceberg format. Ensure that the AWS Glue crawlers are properly configured to update the data catalog and that the Iceberg tables are correctly formatted and updated.

As of now, there are no official announcements from AWS about plans to directly integrate Kinesis Data Firehose with Iceberg formats in AWS Glue. You might want to follow AWS updates or explore using a custom solution involving AWS Lambda or another processing layer to handle data transformation before loading it into Iceberg tables via AWS Glue.

profile picture
專家
已回答 24 天前
profile picture
專家
已審閱 23 天前
  • Thank you. It'd make sense to restrict the schema you can choose from Glue to the basic Hive based tables and omit the Iceberg tables during Firehose -> Parquet schema configuration to lessen the confusion with the customers.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南