AWS Glue Job - Extract the Job related metadata

0

Hi,

we have 50 Glue jobs deployed in our AWS account . I want to be able to extract the job related configuraion metadta as a csv files. The metadata that I want to extract are

  1. script file path
  2. Script File location 3)Job Type
  3. Jar files list
  4. Referenced file Path
  5. Job Language

Is there a command line option or a python script that I can generate to extract this information easily ?

SMR
asked 9 months ago464 views
2 Answers
3
Accepted Answer

I understand that you may be finding little difficulty in filtering the results, specially for default arguments as that contains special characters("--") and to filter those parameters, syntax would be little different. Usual syntax won't work for default arguments.

Here is how the CLI command would look like for your use case:

aws glue get-jobs --query 'Jobs[].[Name,'Command.Name','Command.ScriptLocation','GlueVersion',DefaultArguments."--extra-py-files",DefaultArguments."--job-language",DefaultArguments."--TempDir",DefaultArguments."--extra-files"]' --profile <profile_name> --output table

References:

Glue CLI

CLI Filter Usage

Hope you find this useful.

Abhishek

profile pictureAWS
EXPERT
answered 9 months ago
profile pictureAWS
EXPERT
iBehr
reviewed 9 months ago
0

Hi,

There is a CLI command for this. It is get-jobs: see https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glue/get-jobs.html

You can then use option: --output table to get a tabular format.

Also, you may want to filter only some of the returned fields (they can be abundant with command get-jobs ....) by using --query option: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html

Best,

Didier

profile pictureAWS
EXPERT
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions