CFn and CDK for DeltaTargetProperties

Question

Are there any ways to create a custom DeltaTargetProperties with cdk since it doesn't exist?
I see the big is that the DeltaTargetProperties is not support in CFn yet.

Attempt 1 with docs as code examples
```
  cfn_crawler = glue.CfnCrawler(self, "MyCfnCrawler",
            role=glue_job_crawler_arn,
            targets=glue.CfnCrawler.TargetsProperty(
                delta_targets=glue.CfnCrawler.TargetsProperty(
                    delta_table=[
                       "s3://bucket/dataset_name/table_name/",
                   ],
                   write_manifest = True
                )
            ),
         ...
      )
```

I can run `aws glue get-crawler --name cralwer_name` and get:

```
 "Crawler": {
        "Name": "cralwer_name",
        "Role": "cralwer_role",
        "Targets": {
            "S3Targets": [],
            "JdbcTargets": [],
            "MongoDBTargets": [],
            "DynamoDBTargets": [],
            "CatalogTargets": [],
            "DeltaTargets": [
                {
                    "DeltaTables": [
                         "s3://bucket/dataset_name/table_name/",
                    ],
                    "WriteManifest": true
                }
            ]
        },
        "DatabaseName": "db_name",
        "Classifiers": [],
        "RecrawlPolicy": {
            "RecrawlBehavior": "CRAWL_EVERYTHING"
        },
        "SchemaChangePolicy": {
            "UpdateBehavior": "UPDATE_IN_DATABASE",
            "DeleteBehavior": "LOG"
        },
        "LineageConfiguration": {
            "CrawlerLineageSettings": "DISABLE"
        },
        "State": "READY",
    ...
        "Version": 2,
        "LakeFormationConfiguration": {
            "UseLakeFormationCredentials": false,
            "AccountId": ""
        }
    }

```

Attempt 2 with .... code examples

This loops through a dict with crawler_name as key and crawler_details as value.

```
crawlers[crawler_name]['crawler'] = glue.CfnCrawler(self, "CfnCrawler_" + crawler_name,
                name=crawler_name,
                database_name=crawler_details['database_name'],
                description=crawler_details['description'] + ' Delta Lake Crawler',
                role=self.glue_crawler_role,
                targets={},
                #     'DeltaTargets': [
                #         {
                #             'DeltaTables': crawler_details['delta_tables'],
                #             'WriteManifest': crawler_details['write_manifest']
                #         }
                #     ]
                # },
                schema_change_policy={
                    'UpdateBehavior': 'UPDATE_IN_DATABASE',
                    'DeleteBehavior': 'LOG'
                }
            )

crawlers[crawler_name]['crawler'].add_property_override("Targets.DeltaTargets", [{
        "DeltaTables": crawler_details['delta_tables'],
        "WriteManifest": crawler_details['write_manifest']
    }])
```

❌  CrawlersStack failed: Error: The stack named CrawlersStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Property validation failure: [Encountered unsupported properties in {/Targets}: [DeltaTargets]]
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:505:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at deployStack2 (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:265:24)
    at /usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/deploy.ts:39:11
    at run (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/p-queue/dist/index.js:163:29)

❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named CrawlersStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Property validation failure: [Encountered unsupported properties in {/Targets}: [DeltaTargets]]
    at deployStacks (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/deploy.ts:61:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at CdkToolkit.deploy (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:339:7)
    at exec4 (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cli.ts:384:12)

Answer

It looks like the glue.CfnCrawler class does not currently support the delta_targets property, which is used to specify Delta Lake targets for the Crawler. One workaround for this would be to use the cfn_crawler object's add_property_override method to manually add the DeltaTargets property to the CloudFormation template that is generated by the CDK. For example, you can use the following code to add the DeltaTargets property to your cfn_crawler object:

```
cfn_crawler.add_property_override("Targets.DeltaTargets", [{
    "DeltaTables": [ "s3://bucket/dataset_name/table_name/" ],
    "WriteManifest": True
}])
```

You can also create the resource directly using @aws_cdk.aws_glue.CfnCrawler and pass DeltaTargets property as parameter.

```
glue.CfnCrawler(self, "MyCfnCrawler",
  role=glue_job_crawler_arn,
  targets={
    "DeltaTargets": [
      {
        "DeltaTables": [
          "s3://bucket/dataset_name/table_name/",
        ],
        "WriteManifest": True
      }
    ]
  },
  ...
)
```

CFn and CDK for DeltaTargetProperties

相關內容