Why is there a discrepancy in storage and size metrics between the Amazon S3 console, CloudWatch, and S3 Storage Lens?

Short description

AWS services use different ways to measure storage and report storage values. The CloudWatch metric for daily bucket storage includes incomplete multipart uploads and noncurrent versions. The Amazon S3 Storage Lens metric for total storage includes incomplete multipart uploads, object metadata, and delete markers. The Amazon S3 console metric for total size of each bucket doesn't include incomplete multipart uploads or noncurrent versions.

Resolution

Identify what's causing the difference in storage metrics

To identify the cause of the differences, take the following actions:

• Check whether you turned on S3 Versioning.
• Check whether you have incomplete multipart uploads.
• Check the time span of the report. Because CloudWatch reports metrics for Amazon S3 only once each day, the metrics might not display the most updated information. However, you can configure CloudWatch to monitor your AWS resources and applications in real time.
• Determine whether different base systems caused the calculation differences. The Amazon S3 console and S3 Storage Lens use base 2 conversion (/1024) to report storage metrics. CloudWatch uses base 10 conversion (/1000) by default.

If you turned on S3 Versioning, then use an Amazon S3 inventory list to review your S3 bucket for different object versions. The inventory list captures metadata such as bucket name, object size, storage class, and version ID. You can use Amazon Athena to query the data. You can also use the Current-version bytes and Nonconcurrent version bytes S3 Storage Lens dashboard metrics to calculate the size of the current and noncurrent versions.

To remove incomplete multipart uploads, see Discovering and deleting incomplete multipart uploads to lower Amazon S3 costs. 

Review your storage metrics

Calculate daily storage metrics in CloudWatch

The BucketSizeBytes CloudWatch metric captures all Amazon S3 and Amazon S3 Glacier storage types, object versions, and incomplete multipart uploads. CloudWatch calculates a sum of all object sizes, current and noncurrent objects in your bucket, and incomplete multipart upload sizes. The BucketSizeBytes metric calculates how much data in bytes is in an Amazon S3 bucket in the following object storage classes:

• S3 Standard
• S3 Intelligent-Tiering
• S3 Standard-IA
• S3 One Zone-IA
• S3 Reduced Redundancy Storage
• S3 Glacier Deep Archive
• S3 Glacier Flexible Retrieval
• S3 Glacier Instant Retrieval

The NumberOfObjects CloudWatch metric counts the following values in your bucket:

• The total number of objects among all storage classes. For buckets with versioning, this includes current and noncurrent object versions, and delete markers.
• The total number of parts for incomplete multipart uploads.

For example, if you have two versions of the same object, then the two versions count as two separate objects.

For more information, see Amazon S3 daily storage metrics for buckets in CloudWatch.

Calculate total storage metrics in the S3 Storage Lens dashboard

The Total storage metric is on the Bucket tab. This metric captures incomplete multipart uploads, object metadata, and delete markers.

To view the composition of space that they use, check the following metrics:

• Current version bytes
• Noncurrent version bytes
• Incomplete multipart upload bytes

The preceding metrics are available in AWS Free Tier.

The Object count metric also includes information about current and noncurrent versions, delete markers, and incomplete multipart upload object counts.

For detailed information, check the following metrics:

• Current version object count
• Noncurrent version object count
• Delete marker object count
• Incomplete multipart upload object count

For more information, see Amazon S3 Storage Lens metrics glossary. To set up the S3 Storage Lens dashboard, see Optimize costs and gain visibility into usage with Amazon S3 Storage Lens.

Calculate total bucket size on the Amazon S3 console

On the Amazon S3 console, you can use the Calculate total size action to calculate the size of a bucket. However, total bucket size doesn't include multipart uploads and previous or noncurrent versions. Amazon S3 calculates only the total number of objects for the current or newest version of each object that you store in the bucket. For example, if there are two versions of an object in your bucket, then the Amazon S3 storage calculator counts them as only one object. As a result, the number that the Amazon S3 console calculates is smaller than the one that CloudWatch reports.

Related information

Lifecycle configuration to abort multipart uploads

Expiring objects