We have a glue job that is writing large number of items to dynamo.
If a write to dynamo fails, how can we have access to these individual failed records in order to attempt to resolve and reprocess only these items and not reprocess ALL of the items?
In order to force this behavior to test it - we reduced the dynamo RCUs until the job fails with a "DynamoDB write exceeds max retry".
But some items are still written to dyanmo.
We would like to reprocess only the records that failed - rather than having to reprocess ALL of the records we are attempting to insert/update.
We were hoping to be able to find the error records in errorsAsDynamicFrame() - but those seem to only contains records that had parsing errors when being loaded INTO the dynamic frame. In this case we want to identify the records with errors being written FROM the dynamic frame.
This dynamodb writer functionality seems to be very similar to this project - https://github.com/audienceproject/spark-dynamodb
If the source code is similar - then it seems that what we need is to be able to see the results of TableConnector.HandleBatchWriterResponse.getUnprocessedItems()
Is there any way to get at this information?