CFn and CDK for DeltaTargetProperties

0

Are there any ways to create a custom DeltaTargetProperties with cdk since it doesn't exist? I see the big is that the DeltaTargetProperties is not support in CFn yet.

Attempt 1 with docs as code examples

  cfn_crawler = glue.CfnCrawler(self, "MyCfnCrawler",
            role=glue_job_crawler_arn,
            targets=glue.CfnCrawler.TargetsProperty(
                delta_targets=glue.CfnCrawler.TargetsProperty(
                    delta_table=[
                       "s3://bucket/dataset_name/table_name/",
                   ],
                   write_manifest = True
                )
            ),
         ...
      )

I can run aws glue get-crawler --name cralwer_name and get:

 "Crawler": {
        "Name": "cralwer_name",
        "Role": "cralwer_role",
        "Targets": {
            "S3Targets": [],
            "JdbcTargets": [],
            "MongoDBTargets": [],
            "DynamoDBTargets": [],
            "CatalogTargets": [],
            "DeltaTargets": [
                {
                    "DeltaTables": [
                         "s3://bucket/dataset_name/table_name/",
                    ],
                    "WriteManifest": true
                }
            ]
        },
        "DatabaseName": "db_name",
        "Classifiers": [],
        "RecrawlPolicy": {
            "RecrawlBehavior": "CRAWL_EVERYTHING"
        },
        "SchemaChangePolicy": {
            "UpdateBehavior": "UPDATE_IN_DATABASE",
            "DeleteBehavior": "LOG"
        },
        "LineageConfiguration": {
            "CrawlerLineageSettings": "DISABLE"
        },
        "State": "READY",
    ...
        "Version": 2,
        "LakeFormationConfiguration": {
            "UseLakeFormationCredentials": false,
            "AccountId": ""
        }
    }

Attempt 2 with .... code examples

This loops through a dict with crawler_name as key and crawler_details as value.

crawlers[crawler_name]['crawler'] = glue.CfnCrawler(self, "CfnCrawler_" + crawler_name,
                name=crawler_name,
                database_name=crawler_details['database_name'],
                description=crawler_details['description'] + ' Delta Lake Crawler',
                role=self.glue_crawler_role,
                targets={},
                #     'DeltaTargets': [
                #         {
                #             'DeltaTables': crawler_details['delta_tables'],
                #             'WriteManifest': crawler_details['write_manifest']
                #         }
                #     ]
                # },
                schema_change_policy={
                    'UpdateBehavior': 'UPDATE_IN_DATABASE',
                    'DeleteBehavior': 'LOG'
                }
            )

crawlers[crawler_name]['crawler'].add_property_override("Targets.DeltaTargets", [{
        "DeltaTables": crawler_details['delta_tables'],
        "WriteManifest": crawler_details['write_manifest']
    }])

❌ CrawlersStack failed: Error: The stack named CrawlersStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Property validation failure: [Encountered unsupported properties in {/Targets}: [DeltaTargets]] at FullCloudFormationDeployment.monitorDeployment (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:505:13) at processTicksAndRejections (node:internal/process/task_queues:95:5) at deployStack2 (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:265:24) at /usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/deploy.ts:39:11 at run (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/p-queue/dist/index.js:163:29)

❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named CrawlersStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Property validation failure: [Encountered unsupported properties in {/Targets}: [DeltaTargets]] at deployStacks (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/deploy.ts:61:11) at processTicksAndRejections (node:internal/process/task_queues:95:5) at CdkToolkit.deploy (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:339:7) at exec4 (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cli.ts:384:12)

1 Answer
0

It looks like the glue.CfnCrawler class does not currently support the delta_targets property, which is used to specify Delta Lake targets for the Crawler. One workaround for this would be to use the cfn_crawler object's add_property_override method to manually add the DeltaTargets property to the CloudFormation template that is generated by the CDK. For example, you can use the following code to add the DeltaTargets property to your cfn_crawler object:

cfn_crawler.add_property_override("Targets.DeltaTargets", [{
    "DeltaTables": [ "s3://bucket/dataset_name/table_name/" ],
    "WriteManifest": True
}])

You can also create the resource directly using @aws_cdk.aws_glue.CfnCrawler and pass DeltaTargets property as parameter.

glue.CfnCrawler(self, "MyCfnCrawler",
  role=glue_job_crawler_arn,
  targets={
    "DeltaTargets": [
      {
        "DeltaTables": [
          "s3://bucket/dataset_name/table_name/",
        ],
        "WriteManifest": True
      }
    ]
  },
  ...
)
profile picture
answered a year ago
  • Hey @Divyam, Thanks for jump in and helping out. I added my attempt with your code above in question. Running with CDK version 2.61.1 (build d319d9c)

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions