Google BigQuery Connector for AWS Glue / Request to update to 0.28

Question

The Glue Marketplace Connector "Google BigQuery Connector for AWS Glue" is currently at 0.24.2. Can you please update this connector to 0.28? The newest versions provide direct write using the GRPC streaming write, as well as more forgiving schema handling when using this method. Additionally the current 0.24.2 seems to have issues with passing credentials properly when using indirect write (via cloud storage), which is the only option.

Thanks!

https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases Current connector is: https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/bigquery:0.24.2-glue3.0

Thanks!

https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases
Current connector is: https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/bigquery:0.24.2-glue3.0

Answer

I managed to get this working with a custom connector using spark-bigquery-with-dependencies_2.12-0.28.0.jar
from https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases

Looks like that's all the AWS package is anyways. It's even pulling in my AWS secret ok. Having the AWS connector updated would still be nice.

Google BigQuery Connector for AWS Glue / Request to update to 0.28

관련 콘텐츠