Hi All,
I have some issues when running my glue job, I landed my pipe delimited csv file in a s3 bucket and after running the crawler pointing to the folder where the file is placed, a glue catalog table is created.
However when I tried to read the data(code below) from the catalog table in a glue job for additional processing and converting to parquet, its not picking all the records.
dyf = glueContext.create_dynamic_frame.from_catalog(
database=DATABASE,
table_name=table_name,
transformation_ctx="dyf-" + table_name,
)
rows = dyf.count()
print(f"DataFrame records count : {rows}")
Please can someone suggest what could be the reason for the missing records? I see that there are three columns in the catalog table with incorrect data type( bigint in place of string). I went and manually corrected the data type and set the infer_schema = True in the above code. job is still not picking up the correct number of records.
AWS公式更新しました 3年前
How much is the difference? Is possible you have bookmarks enabled? How many do you get if you do spark.table(f"{DATABASE}.{table_name}").count()
Hi,
Th Glue data Catalog is not a constraint, it is for information. You can force the schema using the glue resolve function. you have to manually force the schema