Skip to content

Audit S3 Lifecycle Policies Org-wide Using AWS Config Aggregator

6 minute read
Content level: Advanced
0

Auditing Amazon S3 lifecycle configuration one bucket at a time doesn't scale, GetBucketLifecycleConfiguration is a per-bucket call with no batch option. This article shows how to use an AWS Config Organization Aggregator with advanced queries to inventory every bucket's lifecycle configuration across all accounts and Regions in an organization, including buckets created in the future. It also covers the correct query patterns that commonly trip people up: S3 sub-configs live under suppleme

Listing S3 Bucket Lifecycle Policies Across All Accounts in an Organization

Using AWS Config Organization Aggregator and Advanced Queries

Auditing Amazon S3 lifecycle configuration one bucket at a time does not scale. The GetBucketLifecycleConfiguration call is per-bucket with no batch option, so scripting it across hundreds or thousands of buckets in a multi-account organization is slow and brittle. A better approach is to let AWS Config do the recording for you: Config captures each S3 bucket's lifecycle configuration as part of its configuration item, and an Organization Aggregator lets you query all of it across every account and Region from one place, including buckets created in the future.

This article walks through the setup and, importantly, the correct way to query the data, since a few common query mistakes will silently return incomplete or empty results.

Architecture: Config Organization Aggregator collecting S3 configuration items from all member accounts


Setup

The setup has two halves: turn on Config recording everywhere, then centralize it in an aggregator you can query.

Step 1 : Enable AWS Config service access for Organizations

Run from the management (payer) account:

aws organizations enable-aws-service-access --service-principal config.amazonaws.com

Step 2 : (Optional) Register a delegated administrator

If you prefer to run this from a dedicated audit or security account rather than the payer, register it as the delegated admin for Config:

aws organizations register-delegated-administrator \
  --account-id <AUDIT_ACCOUNT_ID> \
  --service-principal config.amazonaws.com

Step 3 : Enable AWS Config recording in all member accounts

Deploy this with a CloudFormation StackSet so it applies to current and future accounts. The Config Recorder must record AWS::S3::Bucket, and the Delivery Channel must point at an S3 bucket for snapshots.

Cost control: Config bills per configuration item recorded and per rule evaluation. For a large estate with frequent changes, scope the recorder to the resource types you actually need (for example, AWS::S3::Bucket) rather than recording all resource types.

Step 4 : Create an Organization Aggregator

With recording in place, centralize the data. In the payer or delegated-admin account:

aws configservice put-configuration-aggregator \
  --configuration-aggregator-name "OrgS3Aggregator" \
  --organization-aggregation-source '{"RoleArn":"arn:aws:iam::<ACCT>:role/ConfigRole","AllAwsRegions":true}'

Step 5 : Deploy the managed compliance rule

Finally, add the managed rule that identifies buckets with no lifecycle policy. This becomes the reliable way to answer "which buckets are missing lifecycle rules?" in the query stage.

aws configservice put-organization-config-rule \
  --organization-config-rule-name "s3-lifecycle-check" \
  --organization-managed-rule-metadata '{"RuleIdentifier":"S3_LIFECYCLE_POLICY_CHECK"}'

The identifier S3_LIFECYCLE_POLICY_CHECK applies to AWS::S3::Bucket. It also accepts optional parameters (targetTransitionDays, targetExpirationDays, targetTransitionStorageClass, targetPrefix, bucketNames). Leave them empty for a simple "has any lifecycle rule?" check; set them only if you want to enforce specific rule values.


Querying lifecycle across all accounts

With the aggregator populated, you can query from the SQL editor (Query scope set to the aggregator) or the select-aggregate-resource-config CLI. Three details trip people up, so the queries below account for them:

  1. S3 sub-configurations live under supplementaryConfiguration, not configuration. for example, versioning is supplementaryConfiguration.BucketVersioningConfiguration.status. The values are stored as JSON-encoded strings.
  2. IS NULL is not supported in advanced query, so you cannot use it to find buckets without a lifecycle policy. Use the managed rule's NON_COMPLIANT results instead.
  3. Advanced query does not unpack nested arrays such as lifecycle rules into columns. Select the field, export the result as JSON, and parse it with jq.

Confirm the exact field name first

Field names under supplementaryConfiguration follow a Bucket…Configuration pattern, and the lifecycle key is expected to be BucketLifecycleConfiguration. Confirm it against your own data before relying on it run this, export as JSON, and inspect the keys:

SELECT resourceId, resourceName, supplementaryConfiguration
WHERE resourceType = 'AWS::S3::Bucket'

Pull lifecycle rules for all buckets, all accounts

Once you have the key, select it and flatten the JSON string with jq:

SELECT accountId, awsRegion, resourceId, resourceName,
       supplementaryConfiguration.BucketLifecycleConfiguration
WHERE resourceType = 'AWS::S3::Bucket'
aws configservice select-aggregate-resource-config \
  --configuration-aggregator-name OrgS3Aggregator \
  --expression "SELECT accountId, awsRegion, resourceName, supplementaryConfiguration.BucketLifecycleConfiguration WHERE resourceType = 'AWS::S3::Bucket'" \
  --output json > buckets.json

jq -r '.Results[] | fromjson |
  [.accountId, .awsRegion, .resourceName,
   (.supplementaryConfiguration.BucketLifecycleConfiguration // "NONE")] | @tsv' buckets.json

Results return 100 per page, so paginate with NextToken to cover the whole estate.

Find buckets without a lifecycle policy

Rather than an unsupported IS NULL query, read the compliance results produced by the managed rule from Step 5:

SELECT accountId, awsRegion, resourceId, resourceName
WHERE resourceType = 'AWS::Config::ResourceCompliance'
  AND configuration.complianceType = 'NON_COMPLIANT'
  AND configuration.configRuleList.configRuleName LIKE 's3-lifecycle%'

Organization-deployed rules are often renamed with an OrgConfigRule-… prefix inside member accounts, so matching with LIKE 's3-lifecycle%' is safer than an exact rule name.


Optional: alerting on changes

To stay ahead of drift, notify on non-compliant evaluations with EventBridge and SNS. Use the Config Rules Compliance Change event type, and match on newEvaluationResult.complianceType:

{
  "source": ["aws.config"],
  "detail-type": ["Config Rules Compliance Change"],
  "detail": {
    "messageType": ["ComplianceChangeNotification"],
    "configRuleName": [{ "prefix": "s3-lifecycle" }],
    "newEvaluationResult": { "complianceType": ["NON_COMPLIANT"] }
  }
}

The prefix match on configRuleName handles the OrgConfigRule-… renaming that organization-deployed rules receive inside member accounts. Verify the exact detail fields against a sample event in your own account before relying on the pattern in production.


Summary

TopicDetail
CoverageAll existing and new buckets across every account in the organization
Data locationsupplementaryConfiguration.BucketLifecycleConfiguration (JSON string) not configuration.
Full rulesNot flattenable in SQL (nested array) → export JSON and parse with jq
Missing-lifecycle detectionManaged rule s3-lifecycle-policy-check NON_COMPLIANT results (not IS NULL)
CostPer-configuration-item recording plus per-evaluation charges; scope the recorder accordingly
PaginationAdvanced-query results return 100 per page

References