Does Kinesis Firehose destination support Glue table format of type Iceberg

0

Kinesis Firehose allows to configure S3 as a destination and in the Parquet section allows selecting a Glue catalog table of format Iceberg. However I had little to no luck querying the data.

Does Kinesis Firehose support Glue Iceberg as destination ? Any plans in near future if not.

RegeR
질문됨 24일 전84회 조회
1개 답변
1
수락된 답변

Currently, Kinesis Data Firehose does not natively support direct streaming to AWS Glue with Apache Iceberg table formats as a destination. Kinesis Data Firehose can output to Amazon S3 in formats such as Parquet, which can then be used in conjunction with AWS Glue for further processing and management in Iceberg format. Once the data is in S3, you can utilize AWS Glue to manage and catalog these data into Iceberg format, which supports complex and analytical workloads with robust data management features.

For your specific situation where you had difficulties querying data possibly streamed via Firehose and managed as Iceberg tables in AWS Glue, you might need to check the setup or integration process between S3, AWS Glue, and the Iceberg format. Ensure that the AWS Glue crawlers are properly configured to update the data catalog and that the Iceberg tables are correctly formatted and updated.

As of now, there are no official announcements from AWS about plans to directly integrate Kinesis Data Firehose with Iceberg formats in AWS Glue. You might want to follow AWS updates or explore using a custom solution involving AWS Lambda or another processing layer to handle data transformation before loading it into Iceberg tables via AWS Glue.

profile picture
전문가
답변함 24일 전
profile picture
전문가
검토됨 23일 전
  • Thank you. It'd make sense to restrict the schema you can choose from Glue to the basic Hive based tables and omit the Iceberg tables during Firehose -> Parquet schema configuration to lessen the confusion with the customers.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠