AWS Builder Center: Learn, Build and Connect with builders in the AWS community
AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
Audit S3 Lifecycle Policies Org-wide Using AWS Config Aggregator
Auditing Amazon S3 lifecycle configuration one bucket at a time doesn't scale, GetBucketLifecycleConfiguration is a per-bucket call with no batch option. This article shows how to use an AWS Config Organization Aggregator with advanced queries to inventory every bucket's lifecycle configuration across all accounts and Regions in an organization, including buckets created in the future. It also covers the correct query patterns that commonly trip people up: S3 sub-configs live under suppleme
Listing S3 Bucket Lifecycle Policies Across All Accounts in an Organization
Using AWS Config Organization Aggregator and Advanced Queries
Auditing Amazon S3 lifecycle configuration one bucket at a time does not scale. The
GetBucketLifecycleConfiguration call is per-bucket with no batch option, so scripting it
across hundreds or thousands of buckets in a multi-account organization is slow and brittle.
A better approach is to let AWS Config do the recording for you: Config captures each S3
bucket's lifecycle configuration as part of its configuration item, and an Organization
Aggregator lets you query all of it across every account and Region from one place,
including buckets created in the future.
This article walks through the setup and, importantly, the correct way to query the data, since a few common query mistakes will silently return incomplete or empty results.
Setup
The setup has two halves: turn on Config recording everywhere, then centralize it in an aggregator you can query.
Step 1 : Enable AWS Config service access for Organizations
Run from the management (payer) account:
aws organizations enable-aws-service-access --service-principal config.amazonaws.com
Step 2 : (Optional) Register a delegated administrator
If you prefer to run this from a dedicated audit or security account rather than the payer, register it as the delegated admin for Config:
aws organizations register-delegated-administrator \ --account-id <AUDIT_ACCOUNT_ID> \ --service-principal config.amazonaws.com
Step 3 : Enable AWS Config recording in all member accounts
Deploy this with a CloudFormation StackSet so it applies to current and future accounts. The
Config Recorder must record AWS::S3::Bucket, and the Delivery Channel must point at an S3
bucket for snapshots.
Cost control: Config bills per configuration item recorded and per rule evaluation. For a large estate with frequent changes, scope the recorder to the resource types you actually need (for example,
AWS::S3::Bucket) rather than recording all resource types.
Step 4 : Create an Organization Aggregator
With recording in place, centralize the data. In the payer or delegated-admin account:
aws configservice put-configuration-aggregator \ --configuration-aggregator-name "OrgS3Aggregator" \ --organization-aggregation-source '{"RoleArn":"arn:aws:iam::<ACCT>:role/ConfigRole","AllAwsRegions":true}'
Step 5 : Deploy the managed compliance rule
Finally, add the managed rule that identifies buckets with no lifecycle policy. This becomes the reliable way to answer "which buckets are missing lifecycle rules?" in the query stage.
aws configservice put-organization-config-rule \ --organization-config-rule-name "s3-lifecycle-check" \ --organization-managed-rule-metadata '{"RuleIdentifier":"S3_LIFECYCLE_POLICY_CHECK"}'
The identifier S3_LIFECYCLE_POLICY_CHECK applies to AWS::S3::Bucket. It also accepts
optional parameters (targetTransitionDays, targetExpirationDays,
targetTransitionStorageClass, targetPrefix, bucketNames). Leave them empty for a simple
"has any lifecycle rule?" check; set them only if you want to enforce specific rule values.
Querying lifecycle across all accounts
With the aggregator populated, you can query from the SQL editor (Query scope set to the
aggregator) or the select-aggregate-resource-config CLI. Three details trip people up, so the
queries below account for them:
- S3 sub-configurations live under
supplementaryConfiguration, notconfiguration.for example, versioning issupplementaryConfiguration.BucketVersioningConfiguration.status. The values are stored as JSON-encoded strings. IS NULLis not supported in advanced query, so you cannot use it to find buckets without a lifecycle policy. Use the managed rule's NON_COMPLIANT results instead.- Advanced query does not unpack nested arrays such as lifecycle
rulesinto columns. Select the field, export the result as JSON, and parse it withjq.
Confirm the exact field name first
Field names under supplementaryConfiguration follow a Bucket…Configuration pattern, and the
lifecycle key is expected to be BucketLifecycleConfiguration. Confirm it against your own data
before relying on it run this, export as JSON, and inspect the keys:
SELECT resourceId, resourceName, supplementaryConfiguration WHERE resourceType = 'AWS::S3::Bucket'
Pull lifecycle rules for all buckets, all accounts
Once you have the key, select it and flatten the JSON string with jq:
SELECT accountId, awsRegion, resourceId, resourceName, supplementaryConfiguration.BucketLifecycleConfiguration WHERE resourceType = 'AWS::S3::Bucket'
aws configservice select-aggregate-resource-config \ --configuration-aggregator-name OrgS3Aggregator \ --expression "SELECT accountId, awsRegion, resourceName, supplementaryConfiguration.BucketLifecycleConfiguration WHERE resourceType = 'AWS::S3::Bucket'" \ --output json > buckets.json jq -r '.Results[] | fromjson | [.accountId, .awsRegion, .resourceName, (.supplementaryConfiguration.BucketLifecycleConfiguration // "NONE")] | @tsv' buckets.json
Results return 100 per page, so paginate with NextToken to cover the whole estate.
Find buckets without a lifecycle policy
Rather than an unsupported IS NULL query, read the compliance results produced by the managed
rule from Step 5:
SELECT accountId, awsRegion, resourceId, resourceName WHERE resourceType = 'AWS::Config::ResourceCompliance' AND configuration.complianceType = 'NON_COMPLIANT' AND configuration.configRuleList.configRuleName LIKE 's3-lifecycle%'
Organization-deployed rules are often renamed with an OrgConfigRule-… prefix inside member
accounts, so matching with LIKE 's3-lifecycle%' is safer than an exact rule name.
Optional: alerting on changes
To stay ahead of drift, notify on non-compliant evaluations with EventBridge and SNS. Use the
Config Rules Compliance Change event type, and match on newEvaluationResult.complianceType:
{ "source": ["aws.config"], "detail-type": ["Config Rules Compliance Change"], "detail": { "messageType": ["ComplianceChangeNotification"], "configRuleName": [{ "prefix": "s3-lifecycle" }], "newEvaluationResult": { "complianceType": ["NON_COMPLIANT"] } } }
The prefix match on configRuleName handles the OrgConfigRule-… renaming that organization-deployed
rules receive inside member accounts. Verify the exact detail fields against a sample event in your own
account before relying on the pattern in production.
Summary
| Topic | Detail |
|---|---|
| Coverage | All existing and new buckets across every account in the organization |
| Data location | supplementaryConfiguration.BucketLifecycleConfiguration (JSON string) not configuration. |
| Full rules | Not flattenable in SQL (nested array) → export JSON and parse with jq |
| Missing-lifecycle detection | Managed rule s3-lifecycle-policy-check NON_COMPLIANT results (not IS NULL) |
| Cost | Per-configuration-item recording plus per-evaluation charges; scope the recorder accordingly |
| Pagination | Advanced-query results return 100 per page |
References
- s3-lifecycle-policy-check managed rule
- Querying the current configuration state (advanced query and limitations)
- Resolving advanced-query limitations / nested-array workaround
- S3 supplementaryConfiguration example (GetResourceConfigHistory)
- Multi-account data aggregation
- Using delegated admin for Config aggregation
- Language
- English
Relevant content
- Accepted Answer
asked 4 years ago
