Google BigQuery Connector for AWS Glue / Request to update to 0.28

0

The Glue Marketplace Connector "Google BigQuery Connector for AWS Glue" is currently at 0.24.2. Can you please update this connector to 0.28? The newest versions provide direct write using the GRPC streaming write, as well as more forgiving schema handling when using this method. Additionally the current 0.24.2 seems to have issues with passing credentials properly when using indirect write (via cloud storage), which is the only option.

Thanks!

https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases Current connector is: https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/bigquery:0.24.2-glue3.0

Chris
asked a year ago406 views
1 Answer
0

I managed to get this working with a custom connector using spark-bigquery-with-dependencies_2.12-0.28.0.jar from https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases

Looks like that's all the AWS package is anyways. It's even pulling in my AWS secret ok. Having the AWS connector updated would still be nice.

Chris
answered a year ago
  • Can you elaborate on how did you add this custom connector using the updated version? Did you use the same way as the previous version? I am new to AWS Glue

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions