What event is raised when a Data Migration Task performs DROP_AND_CREATE on a S3

Question

Greetings,

I have an S3 bucket with a Lambda Function notification configured for:

"s3:ObjectCreated:CompleteMultipartUpload",
      "s3:ObjectCreated:Post",
      "s3:ObjectCreated:Put",
      "s3:ObjectRemoved:Delete",

The S3 bucket is also the Target  Endpoint of a Data Migration Task.

I am attempting to intercept every object that is removed from an S3 bucket when a Data Migration Task is re-Started as Full-Load with a TargetTabl;ePreperationMode = DROP_AND_CREATE.

Manually deleting an object from the bucket is properly raising the delete object event notification and the configured Lamba is called with an event source being "s3:ObjectRemoved:Delete", as configured.

Re-starting a Data Migration Task configured as Full-Load with a TargetTabl;ePreperationMode = DROP_AND_CREATE with the target being the S3 bucket does remove all files in the target, however, this does not raise the delete object event notification.

Is "s3:ObjectRemoved:Delete" event raised when a DMS task performs a DROP_AND_CREATE during a full load for each file, or should I be looking for a different event?

Accepted Answer

I found what I think the answer is after trial and error.

I changed the s3 event notification

from "s3:ObjectRemoved:Delete" to  "s3:ObjectRemoved:*",

Now there is an 'eventName': 'ObjectRemoved:DeleteMarkerCreated' being logged to cloudwatch. I am betting that this is the event that I am looking for during the DROP_AND_CREATE phase.

Answer

Amazon S3 can publish events of the following types for Object removal [1]. You specify these event types in the notification configuration.

s3:ObjectRemoved:* 
- You can also use a wildcard s3:ObjectRemoved:* to request notification anytime an object is deleted.

s3:ObjectRemoved:Delete
- You can request notification when an object is deleted or a versioned object is permanently deleted by using the s3:ObjectRemoved:Delete event type.

s3:ObjectRemoved:DeleteMarkerCreated
- Alternatively, you can request notification when a delete marker is created for a versioned object using s3:ObjectRemoved:DeleteMarkerCreated.

Yes, you are correct. Using s3:ObjectRemoved:* event type should handle all the delete event notification since it includes both s3:ObjectRemoved:Delete and s3:ObjectRemoved:DeleteMarkerCreated. It covers the different scenarios such as versioned objects, which i suspect could be the case why the notification didn’t work when manually deleting the objects with an event of s3:ObjectRemoved:Delete.

Just keep in mind that these event notifications don't alert you for automatic deletes from Lifecycle policies or from failed operations.

REFERENCES:
[1]. https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-destinations

What event is raised when a Data Migration Task performs DROP_AND_CREATE on a S3

Relevanter Inhalt