Auditing Amazon QuickSight SPICE Usage: A Python Script to Calculate Total SPICE Consumption Across All Datasets

Managing SPICE capacity in Amazon QuickSight can be challenging, especially when the total usage reported on the Manage QuickSight page doesn't match the sum of your known datasets.

This article presents a Python script using boto3 that:

• Enumerates all SPICE datasets in your account using the ListDataSets API
• Retrieves SPICE size for standard datasets via DescribeDataSet (ConsumedSpiceCapacityInBytes)
• Falls back to ListIngestions for file-based datasets, extracting the size from the most recent successful ingestion (IngestionSizeInBytes)
• Includes a real-time progress indicator and built-in rate limiting to avoid API throttling

Script:

import boto3
import time
import sys

account_id = "<ACCOUNT_ID>"
region = "<REGION>"  # e.g., us-east-1

client = boto3.client("quicksight", region_name=region)

# Step 1: Collect all SPICE dataset summaries
print("Fetching all SPICE datasets...")
paginator = client.get_paginator("list_data_sets")
spice_datasets = []

for page in paginator.paginate(AwsAccountId=account_id):
    for ds in page.get("DataSetSummaries", []):
        if ds.get("ImportMode") == "SPICE":
            spice_datasets.append({
                "id": ds["DataSetId"],
                "name": ds.get("Name", "Unknown")
            })

total_count = len(spice_datasets)
print(f"Found {total_count} SPICE datasets. Now calculating sizes...")

# Step 2: Get size for each dataset
total_spice_bytes = 0
datasets = []
errors = []

for i, ds in enumerate(spice_datasets, 1):
    ds_id = ds["id"]
    ds_name = ds["name"]
    size = 0

    # Progress indicator
    sys.stdout.write(f"\r  Processing {i}/{total_count}: {ds_name[:50]}...")
    sys.stdout.flush()

    # Try DescribeDataSet first
    try:
        detail = client.describe_data_set(
            AwsAccountId=account_id,
            DataSetId=ds_id
        )
        size = detail["DataSet"].get("ConsumedSpiceCapacityInBytes", 0)

    except client.exceptions.InvalidParameterValueException:
        # Fall back to ListIngestions for file-based datasets
        try:
            ing_paginator = client.get_paginator("list_ingestions")
            for ing_page in ing_paginator.paginate(
                AwsAccountId=account_id,
                DataSetId=ds_id
            ):
                for ingestion in ing_page.get("Ingestions", []):
                    if ingestion.get("IngestionStatus") == "COMPLETED":
                        size = ingestion.get("IngestionSizeInBytes", 0)
                        break
                if size > 0:
                    break
        except Exception as e:
            errors.append((ds_name, ds_id, str(e)))

    except Exception as e:
        errors.append((ds_name, ds_id, str(e)))

    size_gb = round(size / (1024**3), 4) if size > 0 else 0
    datasets.append((ds_name, ds_id, size_gb, size))
    total_spice_bytes += size

    # Small delay to avoid API throttling
    time.sleep(0.2)

# Clear the progress line
sys.stdout.write("\r" + " " * 80 + "\r")
sys.stdout.flush()

# Step 3: Display results
datasets.sort(key=lambda x: x[3], reverse=True)

print("" + "=" * 62)
print(f"{'Dataset Name':<45} {'Size (GB)':>15}")
print("-" * 62)
for name, ds_id, size_gb, size_bytes in datasets:
    display_name = name[:43] + ".." if len(name) > 45 else name
    print(f"{display_name:<45} {size_gb:>15}")

total_gb = round(total_spice_bytes / (1024**3), 2)
print("=" * 62)
print(f"{'TOTAL SPICE USAGE':<45} {total_gb:>10} GB")
print(f"{'Total SPICE datasets':<45} {len(datasets):>10}")

if errors:
    print(f"{len(errors)} dataset(s) had errors:")
    print("-" * 62)
    for name, ds_id, err in errors:
        print(f"  - {name} ({ds_id}): {err}")

Result:

Fetching all SPICE datasets...
Found 118 SPICE datasets. Now calculating sizes...
==============================================================           
Dataset Name                                        Size (GB)
-------------------------------------------------------------
customer                                              12.3916
.
.
.
==============================================================
TOTAL SPICE USAGE                                  13.96 GB
Total SPICE datasets                                 118