Automate file sorting after Amazon GuardDuty Malware Protection for S3 scans

Overview

Amazon GuardDuty Malware Protection for Amazon Simple Storage Service (Amazon S3) automatically scans newly uploaded objects in selected S3 buckets for potential malware. After a scan completes, GuardDuty publishes the result to the default Amazon EventBridge event bus with the detail-type GuardDuty Malware Protection Object Scan Result.

This article walks through building an event-driven pipeline that:

1. Receives the scan result event from EventBridge
2. Invokes an AWS Lambda function
3. Copies the object to a clean bucket (if no threats are found) or a quarantine bucket (if threats are found)
4. Deletes the original object from the source bucket

All buckets use server-side encryption with AWS Key Management Service (AWS KMS) keys.

Architecture

Upload Object → [Source Bucket (SSE-KMS)]
                    ↓
        GuardDuty Malware Protection scans object
                    ↓
        EventBridge event published
        (detail-type: "GuardDuty Malware Protection Object Scan Result")
                    ↓
        EventBridge Rule matches event
                    ↓
        Lambda Function invoked
            ├── NO_THREATS_FOUND → copy to Clean Bucket, delete from Source
            └── THREATS_FOUND   → copy to Quarantine Bucket, delete from Source

Prerequisites

• An AWS account with permissions to create IAM roles, S3 buckets, Lambda functions, EventBridge rules, and KMS keys
• AWS CLI v2 installed and configured
• Python 3.12 runtime available for Lambda

Step 1: Create an AWS KMS key

Create a symmetric KMS key for server-side encryption on all three S3 buckets.

aws kms create-key \
  --region YOUR_REGION \
  --description "GuardDuty Malware Protection S3 encryption key" \
  --query 'KeyMetadata.KeyId' --output text

Save the returned key ID. Then create an alias for easier reference:

aws kms create-alias \
  --region YOUR_REGION \
  --alias-name alias/gd-malware-scan-key \
  --target-key-id YOUR_KMS_KEY_ID

Update the key policy to grant the GuardDuty service role and the Lambda execution role permission to use the key. The key policy must include:

• Root account access for key administration
• kms:Decrypt and kms:GenerateDataKey for the GuardDuty role (conditioned on kms:ViaService: s3.YOUR_REGION.amazonaws.com)
• kms:Decrypt, kms:GenerateDataKey, and kms:DescribeKey for the Lambda role

Step 2: Create S3 buckets

Create three buckets: one source bucket (where files are uploaded and scanned), one clean bucket, and one quarantine bucket.

aws s3api create-bucket \
  --bucket YOUR_PREFIX-source-YOUR_ACCOUNT_ID \
  --region YOUR_REGION \
  --create-bucket-configuration LocationConstraint=YOUR_REGION

Repeat for the clean and quarantine buckets with appropriate names.

Important: Turn on versioning on the source bucket. GuardDuty Malware Protection for S3 requires versioning to be active on the protected bucket.

aws s3api put-bucket-versioning \
  --bucket YOUR_PREFIX-source-YOUR_ACCOUNT_ID \
  --region YOUR_REGION \
  --versioning-configuration Status=Enabled

Apply SSE-KMS default encryption to all three buckets with BucketKey turned on. Create a file named encryption-config.json:

{
  "Rules": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "YOUR_KMS_KEY_ID"
      },
      "BucketKeyEnabled": true
    }
  ]
}

aws s3api put-bucket-encryption \
  --bucket YOUR_PREFIX-source-YOUR_ACCOUNT_ID \
  --region YOUR_REGION \
  --server-side-encryption-configuration file://encryption-config.json

Step 3: Create the GuardDuty service role

GuardDuty Malware Protection for S3 requires an IAM role that the service assumes to scan objects. The trust policy must allow the malware-protection-plan.guardduty.amazonaws.com service principal to assume the role.

Create a file named gd-trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "malware-protection-plan.guardduty.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

aws iam create-role \
  --role-name GDMalwareProtectionS3Role \
  --assume-role-policy-document file://gd-trust-policy.json

The permissions policy for this role must include:

• EventBridge managed rule permissions (events:PutRule, events:DeleteRule, events:PutTargets, events:RemoveTargets) scoped to DO-NOT-DELETE-AmazonGuardDutyMalwareProtectionS3* rules
• S3 object tagging permissions on the source bucket
• S3 notification configuration permissions on the source bucket
• S3 read permissions (s3:GetObject, s3:GetObjectVersion) for scanning
• KMS decrypt and generate data key permissions (conditioned on S3 ViaService)

For the complete IAM policy template, see Adding IAM policy permissions in the Amazon GuardDuty User Guide.

Step 4: Turn on GuardDuty Malware Protection for S3

After creating the IAM role, wait approximately 30 seconds for IAM eventual consistency, then create the malware protection plan:

aws guardduty create-malware-protection-plan \
  --region YOUR_REGION \
  --role "arn:aws:iam::YOUR_ACCOUNT_ID:role/GDMalwareProtectionS3Role" \
  --protected-resource '{"S3Bucket":{"BucketName":"YOUR_SOURCE_BUCKET","ObjectPrefixes":["your/prefix/"]}}' \
  --actions '{"Tagging":{"Status":"ENABLED"}}'

The ObjectPrefixes parameter is optional. Use it to limit scanning to objects uploaded under specific prefixes. Turning on Tagging adds a GuardDutyMalwareScanStatus tag to each scanned object.

Verify the plan status:

aws guardduty list-malware-protection-plans --region YOUR_REGION

The status must be ACTIVE. If the status shows WARNING or ERROR, verify the IAM role permissions. For troubleshooting steps, see Troubleshooting Malware Protection plan status.

Step 5: Create the Lambda execution role

Create an IAM role for the Lambda function with a trust policy that allows the lambda.amazonaws.com service principal.

Create a file named lambda-trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}

aws iam create-role \
  --role-name GDScanCopyLambdaRole \
  --assume-role-policy-document file://lambda-trust-policy.json

The permissions policy must grant:

• s3:GetObject, s3:GetObjectVersion, s3:GetObjectTagging, and s3:DeleteObject on the source bucket
• s3:PutObject and s3:PutObjectTagging on the clean and quarantine buckets
• KMS permissions (kms:Decrypt, kms:GenerateDataKey, kms:DescribeKey) on the encryption key
• CloudWatch Logs permissions for logging

Step 6: Deploy the Lambda function

The Lambda function reads the EventBridge event, determines the scan result, and copies the object to the appropriate destination bucket.

This function uses s3.copy() (the boto3 managed transfer method) instead of s3.copy_object(). The managed transfer automatically handles multipart copies for objects larger than the configured threshold, making it suitable for large files up to 5 TB. A TransferConfig object controls multipart behavior and concurrency.

Create a file named lambda_function.py:

import json
import boto3
import urllib.parse
from botocore.client import Config
from boto3.s3.transfer import TransferConfig

# Client config: retry optimization and connection pool for concurrent performance
s3 = boto3.client('s3', config=Config(
    retries={'max_attempts': 3},
    max_pool_connections=20
))

CLEAN_BUCKET = "YOUR_CLEAN_BUCKET"
QUARANTINE_BUCKET = "YOUR_QUARANTINE_BUCKET"

# TransferConfig for large file support and concurrent multipart transfers
# - multipart_threshold: initiate multipart copy for objects larger than 64 MB
# - multipart_chunksize: each part is 64 MB
# - max_concurrency: up to 10 concurrent transfer threads per copy operation
transfer_config = TransferConfig(
    multipart_threshold=64 * 1024 * 1024,
    multipart_chunksize=64 * 1024 * 1024,
    max_concurrency=10,
    use_threads=True
)

def lambda_handler(event, context):
    detail = event.get('detail', {})
    s3_details = detail.get('s3ObjectDetails', {})
    scan_details = detail.get('scanResultDetails', {})

    source_bucket = s3_details.get('bucketName')
    source_key = urllib.parse.unquote_plus(s3_details.get('objectKey', ''))
    scan_status = scan_details.get('scanResultStatus')

    if scan_status == "NO_THREATS_FOUND":
        dest_bucket = CLEAN_BUCKET
    elif scan_status == "THREATS_FOUND":
        dest_bucket = QUARANTINE_BUCKET
    else:
        print(f"Unhandled scan status: {scan_status}")
        return {"statusCode": 200, "body": f"Skipped: {scan_status}"}

    copy_source = {'Bucket': source_bucket, 'Key': source_key}
    s3.copy(copy_source, dest_bucket, source_key, Config=transfer_config)
    s3.delete_object(Bucket=source_bucket, Key=source_key)

    return {
        "statusCode": 200,
        "body": json.dumps({
            "destination": f"s3://{dest_bucket}/{source_key}",
            "scan_status": scan_status
        })
    }

Package and deploy the function. Use 512 MB memory to provide sufficient vCPU for concurrent multipart transfer threads, and increase the timeout to 60 seconds for large file copies:

zip lambda_function.zip lambda_function.py

aws lambda create-function \
  --function-name gd-scan-copy-handler \
  --region YOUR_REGION \
  --runtime python3.12 \
  --role "arn:aws:iam::YOUR_ACCOUNT_ID:role/GDScanCopyLambdaRole" \
  --handler lambda_function.lambda_handler \
  --zip-file fileb://lambda_function.zip \
  --timeout 60 \
  --memory-size 512 \
  --architectures arm64

Step 7: Create the EventBridge rule

Create an EventBridge rule that matches GuardDuty scan result events and targets the Lambda function.

Create a file named event-pattern.json:

{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Malware Protection Object Scan Result"]
}

aws events put-rule \
  --name gd-malware-scan-result-rule \
  --region YOUR_REGION \
  --event-pattern file://event-pattern.json

Add the Lambda function as the rule target:

aws events put-targets \
  --rule gd-malware-scan-result-rule \
  --region YOUR_REGION \
  --targets "Id"="1","Arn"="arn:aws:lambda:YOUR_REGION:YOUR_ACCOUNT_ID:function:gd-scan-copy-handler"

Grant EventBridge permission to invoke the Lambda function:

aws lambda add-permission \
  --function-name gd-scan-copy-handler \
  --region YOUR_REGION \
  --statement-id EventBridgeInvoke \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com \
  --source-arn "arn:aws:events:YOUR_REGION:YOUR_ACCOUNT_ID:rule/gd-malware-scan-result-rule"

Step 8: Test the solution

Upload a clean test file to the source bucket under the monitored prefix:

echo "Hello, this is a clean test file." > /tmp/clean-test.txt

aws s3 cp /tmp/clean-test.txt \
  s3://YOUR_SOURCE_BUCKET/your/prefix/clean-test.txt \
  --region YOUR_REGION

Wait approximately 5-10 seconds for GuardDuty to scan the object and for EventBridge to deliver the event. Then verify:

1. Check the object tag on the source bucket:

aws s3api get-object-tagging \
  --bucket YOUR_SOURCE_BUCKET \
  --key your/prefix/clean-test.txt \
  --region YOUR_REGION

The tag GuardDutyMalwareScanStatus must show NO_THREATS_FOUND.

2. Verify the file was moved to the clean bucket:

aws s3 ls s3://YOUR_CLEAN_BUCKET/your/prefix/ --region YOUR_REGION

3. Confirm the source bucket no longer contains the file:

aws s3 ls s3://YOUR_SOURCE_BUCKET/your/prefix/ --region YOUR_REGION

4. Review Lambda logs for execution details:

aws logs tail /aws/lambda/gd-scan-copy-handler \
  --region YOUR_REGION --since 5m

Considerations

• EventBridge retry behavior: If the Lambda function fails (for example, because of a permission error), EventBridge retries up to 185 times over a maximum of 24 hours using exponential backoff. After you fix the underlying issue, EventBridge automatically retries and processes the object.
• Scan latency: GuardDuty typically completes the scan within a few seconds, but EventBridge delivery adds additional latency. Allow up to 10 seconds before verifying results.
• S3 throttling: The s3Throttled field in the scan result event indicates whether GuardDuty experienced a delay reading the object from S3. If this value is true, it means S3 request rate limits affected the scan. This is rare for typical workloads because S3 supports 3,500 write and 5,500 read requests per second per prefix. It can occur if you upload thousands of objects per second to the same prefix. For more information, see Best practices design patterns: optimizing Amazon S3 performance.
• Skipped scans: Objects might receive a SKIPPED status for reasons such as exceeding the maximum scannable file size. The Lambda function in this article skips those objects. You can extend the logic to handle skipped scans based on your requirements.
• Versioning: The source bucket must have versioning turned on. GuardDuty Malware Protection for S3 requires this configuration.

Cleanup

To avoid ongoing charges, delete the resources in the following order:

1. Delete the GuardDuty Malware Protection plan
2. Remove EventBridge rule targets and delete the rule
3. Delete the Lambda function
4. Delete the Lambda IAM role and its inline policy
5. Delete the GuardDuty IAM role and its inline policy
6. Empty and delete all three S3 buckets (including versioned objects)
7. Delete the KMS key alias and schedule key deletion (minimum 7-day waiting period)
8. Delete the CloudWatch Logs log group

Related resources

• GuardDuty Malware Protection for S3 in the Amazon GuardDuty User Guide
• Monitoring S3 object scans with Amazon EventBridge in the Amazon GuardDuty User Guide
• Adding IAM policy permissions in the Amazon GuardDuty User Guide
• Using Amazon GuardDuty Malware Protection to scan uploads to Amazon S3 on the AWS Security Blog