- Neueste
- Die meisten Stimmen
- Die meisten Kommentare
Hi,
According to AWS documentation, Glue bookmarks are not available for DynamoDB, just for JDBC data sources, and some Amazon S3 sources.
I think you should change node_ddb_sit_planet_payment_merchant.toDF()
to node_ddb_table1.toDF()
to fix this issue.
also you have double check the necessary permissions to write to the specified S3 path.
as stated, since glue doesn't support bookmarks for Dynamo, you can create your own bookmark. all you need to do is use an attribute, like your id field, or a datetime field, probably best to use a epoch (timestamp) field. Then have an index on the ddb table where you can query for values greater than the value from the last run. As a part of your glue job you would store the last processed value in S3, in JSON format for example, and read that value at the start of your glue script.
Relevanter Inhalt
- AWS OFFICIALAktualisiert vor 3 Jahren
- AWS OFFICIALAktualisiert vor 2 Jahren