How can I use the number of CIs recorded per month to understand AWS Config billing?

Resolution

To help you identify previous and current billing trends, use Amazon Athena to identify the number of CIs per month for your account.

Verify that your Amazon S3 bucket contains configuration files

The AWS Config service must be able to successfully deliver configuration history files to your designated Amazon Simple Storage Service (Amazon S3) bucket. AWS Config delivers CIs to the bucket as configuration history files every 6 hours. For more information, see Viewing Compliance History for your AWS Resources with AWS Config.

Complete the following steps:

1. Open the AWS Config console.
2. In the navigation pane, choose Settings.
3. In the Amazon S3 bucket section, note the Bucket name.
4. Open the Amazon S3 console.
5. In the navigation pane, choose Buckets, and then choose your S3 bucket.
6. Review the S3 bucket for configuration files.
   Note: If there are no configuration files, then your role might be missing permissions. For more information, see AWS Identity and Access Management (IAM) for Amazon S3.

Create a table in Athena

Open the Athena console. Then, create a table with the syntax for your environment.

Important: In the following table syntax, replace LOCATION with the location and AWS Region for the AWS Config items stored in your S3 bucket. For BUCKET-NAME, ACCOUNT-ID, REGION, and ORG-ID, use your specific information.

For standard AWS Config implementation

Use the following table syntax:

CREATE EXTERNAL TABLE awsconfig (
         fileversion string,
         configSnapshotId string,
         configurationitems ARRAY < STRUCT < configurationItemVersion : STRING,
         configurationItemCaptureTime : STRING,
         configurationStateId : BIGINT,
         awsAccountId : STRING,
         configurationItemStatus : STRING,
         resourceType : STRING,
         resourceId : STRING,
         resourceName : STRING,
         ARN : STRING,
         awsRegion : STRING,
         availabilityZone : STRING,
         configurationStateMd5Hash : STRING,
         resourceCreationTime : STRING > > 
) 
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' LOCATION 's3://BUCKET-NAME/AWSLogs/ACCOUNT-ID/Config/REGION/';

For Athena engine version 2

If you use Athena engine version 2, then the maximum line length for text files is 100 MB. If you have many resources, then AWS Config items that are stored in the designated S3 bucket might exceed this limit. For example, AWS Config also delivers configuration snapshot files in the same bucket LOCATION, and the configuration snapshot file could exceed this limit. If you exceed the limit, then you receive an error that looks similar to the following when you query the AWS Config item:

"HIVE_BAD_DATA: Line too long in text file: <s3_path_to_config_data_object>"

In this case, use the following table syntax to point Athena to directly query the Amazon S3 path that stores the configuration history files:

CREATE EXTERNAL TABLE awsconfig (
         fileversion string,
         configSnapshotId string,
         configurationitems ARRAY < STRUCT < configurationItemVersion : STRING,
         configurationItemCaptureTime : STRING,
         configurationStateId : BIGINT,
         awsAccountId : STRING,
         configurationItemStatus : STRING,
         resourceType : STRING,
         resourceId : STRING,
         resourceName : STRING,
         ARN : STRING,
         awsRegion : STRING,
         availabilityZone : STRING,
         configurationStateMd5Hash : STRING,
         resourceCreationTime : STRING > > 
) 
PARTITIONED BY (`year` string,`month` string,`day` string)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 's3://BUCKET-NAME/AWSLogs/ACCOUNT-ID/Config/REGION/'
TBLPROPERTIES (
  'projection.enabled'='true', 
  'projection.year.interval'='1', 
  'projection.year.range'='2021,2121', 
  'projection.year.type'='integer', 
  'projection.month.interval'='1', 
  'projection.month.range'='1,12', 
  'projection.month.type'='integer', 
  'projection.day.interval'='1', 
  'projection.day.range'='1,31', 
  'projection.day.type'='integer', 
  'storage.location.template'='s3://BUCKET-NAME/AWSLogs/ACCOUNT-ID/Config/REGION/${year}/${month}/${day}/ConfigHistory/')

Note: The following example table partitions the Athena table with partitions projection from /2021/1/1/ to /2121/12/31/. Update these partitions based on your requirements. The date format of the AWS Config data Amazon S3 path isn't compatible with the date format of the Athena partitions projection.

For AWS Control Tower environments

If you use AWS Control Tower, then to accommodate the account structure, use the following table syntax to create the Athena table:

CREATE EXTERNAL TABLE awsconfig (
       fileversion string,
       configSnapshotId string,
       configurationitems ARRAY < STRUCT < configurationItemVersion : STRING,
       configurationItemCaptureTime : STRING,
       configurationStateId : BIGINT,
       awsAccountId : STRING,
       configurationItemStatus : STRING,
       resourceType : STRING,
       resourceId : STRING,
       resourceName : STRING,
       ARN : STRING,
       awsRegion : STRING,
       availabilityZone : STRING,
       configurationStateMd5Hash : STRING,
       resourceCreationTime : STRING > >
)
PARTITIONED BY (account string,region string,year string,month string,day string)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 's3://BUCKET-NAME/ ORG-ID/AWSLogs/' -- Bucket name and org should be updated
TBLPROPERTIES (
 'projection.enabled'='true',
 'projection.account.type'='enum',
 'projection.account.values'='<account-id1, accountid2>', -- Enter all account ids
 'projection.region.type'='enum',
 'projection.region.values'='<region-1,region-2,region-3>', -- Enter all regions
 'projection.year.interval'='1',
 'projection.year.range'='2021,2121',
 'projection.year.type'='integer',
 'projection.month.interval'='1',
 'projection.month.range'='1,12',
 'projection.month.type'='integer',
 'projection.day.interval'='1',
 'projection.day.range'='1,31',
 'projection.day.type'='integer',
 'storage.location.template'='s3://BUCKET-NAME/ORG-ID/AWSLogs/${account}/Config/${region}/${year}/${month}/${day}/ConfigHistory/') -- Bucket name and org should be updated

Example Athena queries

The following example query retrieves the number of changes per resource in February 2021, sorted by most frequently changed:

SELECT configurationItem.resourceType,
         configurationItem.resourceId,
         COUNT(configurationItem.resourceId) AS NumberOfChanges
FROM default.awsconfig
CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
WHERE "$path" LIKE '%ConfigHistory%'
        AND configurationItem.configurationItemCaptureTime >= '2021-02-01T%'
        AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%'
GROUP BY  configurationItem.resourceType, configurationItem.resourceId
ORDER BY  NumberOfChanges DESC

When you exclude certain resource types from the AWS Config Recorder, the Recorder still records the creation and deletion of those resources. These recordings are done at no cost. These resource CIs include the following statuses for the configurationItemStatus field:

• ResourceNotRecorded
• ResourceDeletedNotRecorded

The recordings also include basic metadata, such as the resource-id, ARN, and CI, but don't include resource configuration and relationship information.

Note: This query still counts the ResourceNotRecorded information. To exclude this information, run the following query:

SELECT configurationItem.resourceType,
       configurationItem.resourceId,
         COUNT(configurationItem.resourceId) AS NumberOfChanges
FROM default.awsconfig
CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
WHERE "$path" LIKE '%ConfigHistory%'
        AND configurationItem.configurationItemCaptureTime >= '2021-02-08T%'
        AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%'
        AND configurationItem.configurationItemStatus NOT IN ('ResourceNotRecorded','ResourceDeletedNotRecorded')
GROUP BY  configurationItem.resourceType
ORDER BY  NumberOfChanges DESC

The results are similar to the following example:

resourcetype              resourceid        NumberOfChanges
AWS::EC2::VPC             vpc-9ed00bfa        7
AWS::EC2::Subnet          subnet-4472e248     5
AWS::EC2::SecurityGroup   sg-450c6531         4

The following example query retrieves the number of configuration items per day in February 2021:

SELECT result.configurationitemcapturetime,
         count(result.configurationitemcapturetime) AS NumberOfChanges
FROM 
    (SELECT regexp_replace(configurationItem.configurationItemCaptureTime,
         '(.+)(T.+)', '$1') AS configurationitemcapturetime
    FROM default.awsconfig
    CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
    WHERE "$path" LIKE '%ConfigHistory%'
            AND configurationItem.configurationItemCaptureTime >= '2021-02-01T%'
            AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%') result
GROUP BY  result.configurationitemcapturetime
ORDER BY  result.configurationitemcapturetime

Note: To filter out non-recorded resources, you can add the following additional clause before the GROUP BY option:

"AND configurationItem.configurationItemStatus NOT IN ('ResourceNotRecorded','ResourceDeletedNotRecorded')"

The results are similar to the following example:

configurationitemcapturetime    NumberOfChanges
2021-02-02    7
2021-02-03    3
2021-02-07   11
...

Note: When you compare the number of CIs between Athena query results and AWS billing data for the same month and Region, a discrepancy can occur. The data that Athena queries can cross day boundaries and include CIs billed in adjacent months. AWS Config CIs are metered based on when you initiated configurationItemCaptureTime.

As a best practice, increment the specified end day from end of the month by one.

For example, you can change the original query to the following updated query.

Original query:

AND configurationItem.configurationItemCaptureTime <= '2021-02-28T%') result

Updated query:

AND configurationItem.configurationItemCaptureTime <= '2021-03-01T%') result

Note: The last day of the month is incremented by one day.

Related information

AWS Config pricing

Amazon Athena pricing

Recording AWS Resources with AWS Config