Glue connection (mysql) issue with whl file, glue job now getting initialize.

Question

I am trying to use WHL file which contains all the packages and glue connection for etl job, The job is not getting initiliaze and is not populating or creating the logs and job run remains in running state. I am using ---additional-python-modules to install my whl file. Though the connection is established and I have tested connection successfully using glue data catalog connection. when I don't use whl file logs get generated, but for whl file it doesn't work and remains in running state.

Also, if I use zip file instead of whl file for utilizing my package along with glue connection(mysql) it gets executed successfully. Please provide solution for utilizing whl file along with glue connection

Also, if I use zip file instead of whl file for utilizing my package along with glue connection(mysql) it gets executed successfully.
Please provide solution for utilizing whl file along with glue connection

Answer

There is a specific section on the Glue Documentation about this.
* Make sure the WHl file path is correct and accessible to the job. You can check CloudWatch logs for any errors related to loading the WHl file.
* Try installing the WHl packages separately using `pip install whl_file_path.whl` on the job environment before running the job code. This will ensure the packages are available.
* Compare the job configuration and role permissions when using the WHl vs zip file. The role may be missing permissions for the WHl file location.
* As a test, try creating a simple Python file without any Glue connection code, just import statements for packages in the WHl. Run this file on the job to isolate if the issue is with package installation or Glue connection code.

https://repost.aws/knowledge-center/glue-version2-external-python-libraries

Glue connection (mysql) issue with whl file, glue job now getting initialize.

相關內容