Data in partitions on S3 bucket, getting below error in preview data query:

Question

HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'site_rank' in table 'luci-data.luci-newslinkdata' is declared as type 'double', but partition 'partition_0=2011-12-30' declared column 'site_rank' as type 'bigint'.
then, I followed the steps mentioned in: 
repost.aws/questions/QU1vPk1069Q5qg4iyuQWTk6Q/while-importing-s3-data-into-quicksight-from-glue-database-getting-the-following-error#ANMuJykKnKSD-jKoZk2tbvqw
after updating in glue crawler: Update all new and existing partitions with metadata from the table
Now the error is as follows:
HIVE_BAD_DATA: Field site_rank's type INT64 in parquet file s3://rnvd-s3-luci-poc/NewsLinkData/2011-06-26/output_4404.parquet is incompatible with type double defined in table schema

Answer

What you are trying to do is not allowed, you cannot just change the types in the metadata if they don't match (or are compatible) with the actual data files.  
If you don't want to update the data files, you can read as bigint and then cast in your query.   
If you have mixture of parquet files with both types, the table is broken, replace/delete one of the set of files.

Data in partitions on S3 bucket, getting below error in preview data query:

相關內容