AWS Glue BigQuery Connection Launch Error: atleast spark.hadoop.google.cloud.auth.service.account.json.keyfile OR credentials must be present in secretId Please refer logs for details.

Question

I am trying to create a Glue job for migrating data from BigQuery to an S3 bucket. I have created a BigQuery connection in AWS Glue. The connection uses the Secret Id to reference a secret name in AWS Secrets Manager, which contains the necessary credentials for connecting to BigQuery. However, when I try to run the job, I receive the following error: 
```
LAUNCH ERROR | atleast spark.hadoop.google.cloud.auth.service.account.json.keyfile OR credentials must be present in secretIdPlease refer logs for details.
```
I'm not sure how to resolve this issue. Any help in resolving this issue would be greatly appreciated. Thanks!

Answer

Hello good morning,

Thank you for your question. Regarding connections in AWS Glue for BigQuery, this tutorial explains how to create a AWS Glue connection for BigQuery to be used in your AWS Glue job.

Link: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-bigquery-home.html

Besides, the link below is a video from AWS channel in YouTube that contains the same architecture you want to create in your AWS Account, explaining the step by step (specially AWS Secrets Manager).

Link: https://www.youtube.com/watch?v=n6fkX5LpEYY

Please me know the results.

AWS Glue BigQuery Connection Launch Error: atleast spark.hadoop.google.cloud.auth.service.account.json.keyfile OR credentials must be present in secretId Please refer logs for details.

Relevant content