AWS Glue Job parameter --enable-data-lineage

0

While running Glue I see these arguments passed to job:

{ 'job_bookmark_option': 'job-bookmark-disable', 'job_bookmark_from': None, 'job_bookmark_to': None, 'JOB_ID': 'j_c8afc16edb1420c2fb878249843e27280db60efcd37b4f6c7c469c4a55a1b5bd', 'JOB_RUN_ID': 'jr_d74caf9a56f744d09ac4d7fd076caa3d8da3cbc5d58f925ea532dc3c7dfcdf32', 'SECURITY_CONFIGURATION': None, 'encryption_type': None, 'enable_data_lineage': None, 'RedshiftTempDir': 's3://aws-glue-assets-myaccount-us-east-1/temporary/', 'TempDir': 's3://aws-glue-assets-myaccount-us-east-1/temporary/', 'JOB_NAME': 'my-job', }

I spotted parameter called "enable_data_lineage" . For the next run I set this parameter to True, like this: "--enable-data-lineage true" in Job parameters section.

After this my job startup time jumped from 7 seconds to 3 min and 10 seconds. I went to logs to check what's going on and I spotted error messages like this:

2022-12-30 12:27:54,873 WARN [Thread-12] lineage.LineagePersistence$ (LineagePersistence.scala:isCatalogLineageSettingEnabled(99)): Exception occurred while getting catalog lineage settings, lineage for this job run will be disabled com.amazonaws.services.lakeformation.model.InternalServiceException: Received an unexpected Content-Type: text/html', expected one of [application/json]. HTTP status code:503 (Service: AWSLakeFormation; Status Code: 500; Error Code: InternalServiceException; Request ID: ffe50f30-ec28-4648-814a-e267be0453da; Proxy: null)

I tried to search for documentation, but no luck..

How to properly set-up this feature? Is there any examples?

posta un anno fa703 visualizzazioni
1 Risposta
1
Risposta accettata

Hi, that property is part of a preview feature not currently available. It should not be used or set to true. that is the reason why it is not documented.

hope this helps,

AWS
ESPERTO
con risposta un anno fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande