Introducing Automatic replication after tagging for Amazon S3

If you replicate Amazon S3 objects based on object tags, you have likely noticed that tagging an object after upload does not trigger replication. Amazon S3 Replication acts on an object at two points, and only two: when the object is written, and when an S3 Batch Replication job includes that object version in its manifest. Tagging an existing object is neither. The object matches your tag-scoped rule, but nothing re-evaluates it.

Today we're sharing Automatic replication after tagging for Amazon S3, a source-side solution that closes that gap. It reads tagging activity from the bucket's S3 Metadata journal table and submits Batch Replication jobs for the objects that now match. To get started, see github.com/aws-samples/sample-replicate-on-tag-for-amazon-s3.

Important: Automatic replication after tagging for Amazon S3 is sample code provided for demonstration and educational purposes. It is not intended for production use without thorough testing, security review, and validation for your specific use case and environment. You are responsible for evaluating whether this solution meets your requirements before deploying it.

The Problem

Tag-scoped rules look like the right tool for workflows where an object's eligibility is not known when it is uploaded. A malware scanner marks an object clean once the scan finishes. A classification job marks a file non-sensitive, sometimes days later, once a reviewer has signed it off. Either way the tag arrives after the PUT, and replication has already made its decision.

Tagging is also a natural selection mechanism. When a dataset that already exists in one Region is needed in another, close to the compute that will process it, tagging is a direct way to nominate the subset that has to move. Here again the objects were written long before the tag was applied.

Two workarounds exist today. Copying an object in place triggers replication, but it re-uploads the data and resets the object's metadata. Copying the objects to the destination yourself avoids the re-upload, but what arrives is a new object, with a new version ID and last-modified time.

How It Works

The solution runs on a schedule, every 15 minutes by default. On each run, for each source bucket, it queries the S3 Metadata journal table for tagging activity since its last checkpoint, matches those objects against the tag-scoped rules already on the bucket, and submits one Batch Replication job covering every match. Rules with no tag filter are ignored.

The checkpoint advances only when a job is submitted successfully, so a run that fails part way through resumes rather than skips ahead.

Everything happens on the source side. The solution never accesses the destination account or Region, and none of the roles it creates holds a destination-side permission, which suits environments where another team administers the destination.

What Makes It Different

The solution submits Batch Replication jobs instead of copying objects itself. Replication reproduces the source object's system metadata on the destination, including the original last-modified time and version ID, and carries the object tags across. A copy writes a new object with a new last-modified time and version ID. Per the AWS Storage blog, replication is the only S3 method that preserves the source last-modified time, which means destination-side lifecycle rules and age-based reporting behave as though the data had always been there. Batch Replication also has no 5 GB per-object limit, so large objects need no custom code.

	S3 Batch Operations Copy	Batch Replication
Last-modified time	New value on copy	Preserved from source
Version ID	New version ID	Preserved from source
ETag	May differ	Matches source
Object tags	Copied only if requested	Replicated
Maximum object size	5 GB, larger needs custom code	50 TB

It is also journal-driven, not event-driven. The journal table is an ordered, queryable record of every tagging operation on the bucket, delivered by S3 into a fully managed Apache Iceberg table in Amazon S3 Tables, which gives the solution a durable position to resume from. There is no per-object Lambda invocation, and no window in which a dropped event leaves an object unreplicated. A burst larger than one run can process drains across runs rather than being dropped: the row cap always reserves capacity for new operations, so both the backlog and new tagging keep moving. JournalLookbackSeconds protects against late-arriving journal records.

Deploying it requires no changes to any role you already own, because it creates the role its jobs run as. Batch Replication uses two roles: a job role that initiates replication, and your bucket's replication configuration role that performs the delivery. The stack creates the job role and scopes it to the buckets you name. Required AWS Permissions covers both roles and the purpose of each grant.

Confirming Replication Finished

A Batch Replication task's status in the completion report reflects whether the object actually replicated, not just whether replication was initiated. Each terminal job with at least one invoked task writes an S3 Batch Operations completion report, and the solution reads it directly to log the error code, such as InitiateReplicationNotPermitted, with the number of objects affected. Setting CompletionNotificationEmail sends one grouped Amazon SNS report per source bucket once each job's report is read.

If a bucket's jobs keep failing, the solution disables that bucket and continues with the others, which caps the per-job charges a failing bucket can incur. Re-enabling it is a one-field edit on the State Bucket, with no redeploy.

Every run emits structured JSON logs, and setting MetricsNamespace publishes per-bucket counters to Amazon CloudWatch covering what each bucket read, matched, and submitted, along with whether it errored.

Replicating Only Malware-Scanned Objects

Amazon GuardDuty Malware Protection for S3 scans newly uploaded objects and, with object tagging enabled, adds GuardDutyMalwareScanStatus:<scan result> to each object once the scan finishes. GuardDuty applies that tag after the object is written, so replication never sees it. Pairing GuardDuty with this solution replicates only the objects GuardDuty has confirmed clean.

1. Enable Malware Protection for S3 on the source bucket with tagging turned on, before objects are uploaded. GuardDuty cannot tag an object whose scan has already run.
2. Add a tag-scoped replication rule whose filter is GuardDutyMalwareScanStatus = NO_THREATS_FOUND.
3. Deploy the solution with the bucket in SourceBucketNames.

Objects tagged THREATS_FOUND, or any other non-clean value, do not match the rule and are not replicated. Use cases covers the dataset-selection pattern and how to fan out to several destination buckets by tag value.

Cost

The solution uses pay-per-use services only, with no fixed or idle charges. S3 Batch Operations is usually the largest component, at a per-job plus per-object charge, with one job per source bucket per run that finds matches. Rule count does not affect cost.

For one source bucket tagging 10,000 objects a day, the worked example in Cost detail comes to roughly $13 a month checking hourly and $14 checking every 15 minutes. The job charge does not vary with the check interval; only Lambda scales with the number of runs.

Getting Started

The solution is available at github.com/aws-samples/sample-replicate-on-tag-for-amazon-s3, as a CloudFormation template plus a Lambda package. Per source account and Region:

1. Ensure S3 Metadata journal table is enabled on each source bucket.
2. Download template.yaml and package-<version>.zip from the Releases page.
3. Upload the zip to an S3 bucket in the same Region as the stack. Its S3 URI, for example s3://amzn-s3-demo-code-bucket/package-<version>.zip, is the CodeLocation parameter.
4. Create the stack, set CodeLocation and SourceBucketNames, and acknowledge CAPABILITY_IAM.

Those two parameters are the only required ones; everything else has a default. The deployment guide has the full parameter reference.

Reminder: this is sample code. Review Required AWS Permissions, the hardening options that are off by default, and the cost implications before deploying into production. Feedback, issues, and pull requests welcome.