How to filter by file type in a Glue table?

I have an S3 bucket with mixed files (CSV and JSON), written there by an external process that is outside of my control.

I need to create two Glue tables:

Contains all of the data from CSVs
Contains all of the data from JSONs

Is this possible?

主題

分析

標籤

AWS Glue

語言

English

m0ltar

已提問 3 個月前檢視次數 161 次

1 個回答

最新
最多得票
最多評論

這些答案是否有幫助？支持正確答案，以幫助社區從您的知識中受益。

The only standard way I know to do that is creating a symlink table (it has a manifest that points to the specific files, which you would need to update). But normally you would want to convert to columnar formats, standardize, transform, partitions or do other things before you create the table, take that chance to separate the files and generate good tables. I would save my users the pain of using CSV/JSON tables.

專家

Gonzalo Herreros

已回答 3 個月前

m0ltar
3 個月前

The only standard way I know to do that is creating a symlink table (it has a manifest that points to the specific files, which you would need to update).

Could you please elaborate on this. I am not too familiar with the concept of a "symlink table".

But normally you would want to convert to columnar formats, standardize, transform, partitions or do other things before you create the table

And to do that, we want to use Athena, potentially via dbt. The idea was to define base raw tables, with partitions, then compact the data using Athena.
Gonzalo Herreros 專家
3 個月前
https://athena.guide/articles/stitching-tables-with-symlinktextinputformat

How to filter by file type in a Glue table?

相關內容