How can I understand AWS Config billing by retrieving the number of configuration items recorded per month?
I want the number of configuration items recorded by AWS Config.
Resolution
To help you identify previous and current billing trends, use Amazon Athena to identify the number of configuration items (CI) per month for your account.
Verify that your S3 bucket contains configuration files
Note: Be sure that the AWS Config service is able to deliver configuration history files successfully to your designated Amazon Simple Storage Service (Amazon S3) bucket. Typically, CIs are delivered to the bucket as configuration history files every 6 hours. For more information, see Viewing configuration details.
- Open the AWS Config console, and then choose Settings from the navigation pane.
- In the Amazon S3 bucket section, note the Bucket name.
- Open the S3 console. Then, for Bucket name, choose your S3 bucket. Verify that the S3 bucket you choose contains configuration files.
Note: If there are no configuration files, then your role might be missing permissions. For more information, see Identity and access management in Amazon S3.
Create a table in Athena
- Sign in to the Athena console, and then follow the instructions to create a table using the wizard.
- Use the following table syntax:
CREATE EXTERNAL TABLE awsconfig ( fileversion string, configSnapshotId string, configurationitems ARRAY < STRUCT < configurationItemVersion : STRING, configurationItemCaptureTime : STRING, configurationStateId : BIGINT, awsAccountId : STRING, configurationItemStatus : STRING, resourceType : STRING, resourceId : STRING, resourceName : STRING, ARN : STRING, awsRegion : STRING, availabilityZone : STRING, configurationStateMd5Hash : STRING, resourceCreationTime : STRING > > ) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' LOCATION 's3://<BUCKET-NAME>/AWSLogs/<ACCOUNT-ID>/Config/<REGION>/';
For LOCATION, use the location and Region for the AWS Config items stored in your Amazon S3 bucket. For BUCKET-NAME, ACCOUNT-ID, and REGION, use your specific information.
Note: If you're using Athena engine version 2, then the maximum line length for text files is 100 MB. If you have a large number of resources, then Config items stored in the designated AWS Config S3 bucket might exceed this limit. For example, AWS Config also delivers configuration snapshot files in the same bucket LOCATION, and the configuration snapshot file could exceed this limit. If you exceed the limit, then when querying the AWS Config item, you receive an error that looks similar to the following:
HIVE_BAD_DATA: Line too long in text file: <s3_path_to_config_data_object>
In this case, use the following table syntax to point Amazon Athena to directly query the S3 path that stores the configuration history files instead:
CREATE EXTERNAL TABLE awsconfig ( fileversion string, configSnapshotId string, configurationitems ARRAY < STRUCT < configurationItemVersion : STRING, configurationItemCaptureTime : STRING, configurationStateId : BIGINT, awsAccountId : STRING, configurationItemStatus : STRING, resourceType : STRING, resourceId : STRING, resourceName : STRING, ARN : STRING, awsRegion : STRING, availabilityZone : STRING, configurationStateMd5Hash : STRING, resourceCreationTime : STRING > > ) PARTITIONED BY (`year` string,`month` string,`day` string) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' LOCATION 's3://<BUCKET-NAME>/AWSLogs/<ACCOUNT-ID>/Config/<REGION>/' TBLPROPERTIES ( 'projection.enabled'='true', 'projection.year.interval'='1', 'projection.year.range'='2021,2121', 'projection.year.type'='integer', 'projection.month.interval'='1', 'projection.month.range'='1,12', 'projection.month.type'='integer', 'projection.day.interval'='1', 'projection.day.range'='1,31', 'projection.day.type'='integer', 'storage.location.template'='s3://<BUCKET-NAME>/AWSLogs/<ACCOUNT-ID>/Config/<REGION>/${year}/${month}/${day}/ConfigHistory/')
For LOCATION, use the location and Region for the AWS Config items stored in your Amazon S3 bucket. For BUCKET-NAME, ACCOUNT-ID, and REGION, use your specific information. The following example table partitions the Athena table with partitions projection from /2021/1/1/ to /2121/12/31/. Customize this time period as needed.
Note: The AWS Config data S3 path date format isn't compatible with Athena partitions projection date type format.
Example Athena queries
The following example query retrieves the number of configuration items per day in February 2021:
SELECT result.configurationitemcapturetime, count(result.configurationitemcapturetime) AS NumberOfChanges FROM (SELECT regexp_replace(configurationItem.configurationItemCaptureTime, '(.+)(T.+)', '$1') AS configurationitemcapturetime FROM default.awsconfig CROSS JOIN UNNEST(configurationitems) AS t(configurationItem) WHERE "$path" LIKE '%ConfigHistory%' AND configurationItem.configurationItemCaptureTime >= '2021-02-01T%' AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%') result GROUP BY result.configurationitemcapturetime ORDER BY result.configurationitemcapturetime
The results are similar to:
configurationitemcapturetime NumberOfChanges 2021-02-02 7 2021-02-03 3 2021-02-07 11 ...
The following example query retrieves the number of changes per resource in February 2021, sorted by most frequently changed:
SELECT configurationItem.resourceType, configurationItem.resourceId, COUNT(configurationItem.resourceId) AS NumberOfChanges FROM default.awsconfig CROSS JOIN UNNEST(configurationitems) AS t(configurationItem) WHERE "$path" LIKE '%ConfigHistory%' AND configurationItem.configurationItemCaptureTime >= '2021-02-01T%' AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%' GROUP BY configurationItem.resourceType, configurationItem.resourceId ORDER BY NumberOfChanges DESC
The results are similar to:
resourcetype resourceid NumberOfChanges AWS::EC2::VPC vpc-9ed00bfa 7 AWS::EC2::Subnet subnet-4472e248 5 AWS::EC2::SecurityGroup sg-450c6531 4
Note: When comparing the total number of CIs between Athena query results and AWS billing data for the same month and Region, a discrepancy can occur. The data queried by Athena can cross day boundaries and also include CIs billed in adjacent months. AWS Config CIs are metered based on when the time configurationItemCaptureTime was initiated.
As a best practice, increment the specified end day from end of month by one.
For example, change this query:
AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%') result
To this query:
AND configurationItem.configurationItemCaptureTime <= '2021-03-01T%') result
Note: The last day of the month is incremented by one day.