Skip to content

CloudWatch Now Speaks PromQL. Do You Still Need Prometheus?

18 minute read
Content level: Advanced
0

Should I use AMP or CloudWatch now that CloudWatch supports PromQL? Until June the answer was short: Prometheus semantics on AWS meant Amazon Managed Service for Prometheus (AMP), and CloudWatch was where AWS service metrics lived.

This year CloudWatch started ingesting OpenTelemetry metrics over OTLP, querying them with PromQL, and, since July, scraping your exporters with managed collectors. This post covers where the two differ, how each one is billed, and how to start on either (or both), for any compute type with a /metrics endpoint.

Rates for both services change, so price your own workload on the CloudWatch pricing page and the AMP pricing page.

The four pieces

CloudWatch OpenTelemetry metrics with PromQL

Next to classic metrics (namespaces, up to 30 dimensions, PutMetricData and EMF, billed per metric per month), CloudWatch now has OpenTelemetry metrics, which AWS labels the recommended path for new workloads. It went GA in June 2026 (What's New) in all commercial Regions except Middle East (UAE), Middle East (Bahrain), and Israel (Tel Aviv). You send to a Regional OTLP/HTTP endpoint, https://monitoring.<region>.amazonaws.com/v1/metrics, and query with PromQL (Prometheus 3.0 specification, including UTF-8 names) in Query Studio or through a Prometheus-compatible HTTP API.

Supported types are gauge, sum, histogram, and exponential histogram. If your exporters expose summaries (common with JMX exporters and Java clients), check in Query Studio what actually lands.

The two models don't share query APIs. GetMetricData, ListMetrics, and Metrics Insights only see classic metrics. The PromQL API only sees OTel metrics (Metrics in Amazon CloudWatch). OTel enrichment (below) is what brings AWS vended metrics into PromQL.

Amazon Managed Service for Prometheus workspaces

A workspace is an isolated Prometheus-compatible store with its own remote write endpoint, query endpoint, ruler, and Alertmanager. You ingest with Prometheus remote write, from Prometheus in agent mode, an ADOT or OpenTelemetry Collector, or an AMP managed collector (ingest methods).

Managed collectors: one API, two destinations

The "CloudWatch managed Prometheus collector" and the "AMP managed scraper" are the same feature with the same scrape configuration and a different destination. Both are created with the CreateScraper API (aws amp create-scraper, even when the destination is CloudWatch), which sends to either an AMP workspace or a CloudWatch dataset (create-scraper CLI reference). Only the destination block differs. For AMP:

{ "ampConfiguration": { "workspaceArn": "arn:aws:aps:us-east-1:123456789012:workspace/ws-EXAMPLE" } }

For CloudWatch:

{ "cloudWatchConfiguration": { "datasetArn": "arn:aws:cloudwatch:us-east-1:123456789012:dataset/default" } }

Every account already has a default CloudWatch dataset in each Region. The CloudWatch destination launched on July 31, 2026 and is available wherever the CloudWatch OTLP endpoint is, except Asia Pacific (New Zealand), ap-southeast-6 (listed as "Asia Pacific (Auckland)" in the PromQL Regions table).

Either flavor is charged by the hour on top of the ingestion charge at its destination. Rates are on the CloudWatch and AMP pricing pages.

Side by side: where they differ

CloudWatch OTel metricsAmazon Managed Service for Prometheus
Ingestion protocolOTLP over HTTP 1.1 only (no gRPC)Prometheus remote write
Ingestion meteringBytes of uncompressed OTLP payload ingestedMetric samples ingested, tiered by volume within a Region
Storage15 months includedBilled separately; 150 days default retention, up to 1,095 days
Query meteringConsole (Query Studio, dashboards): free. PromQL API: samples scannedQuery samples processed
ScalePer account: 500 requests/s; 1,000 data points and 5 MB uncompressed per request; 1M new series per 10 minutes50M active series default, up to 1.5B per workspace; ~1.67M samples/s per workspace (adjustable)
Late dataTimestamps up to 14 days in the past, 2 hours in the futureSamples older than 1 hour refused; out-of-order window 60 s default, up to 600 s
Query limits500 series returned per query; 7-day max range per request; 100,000 series and 300M samples scanned per 24 h window95-day max range; 12M series fetched and 50M samples per 24-hour interval of a single query
HistogramsAll histograms stored as exponential histograms (explicit-bucket histograms are converted at ingestion); delta and cumulativeClassic histograms plus Prometheus native histograms (GA June 2026)
Rules and alertingPromQL alarms (per-series contributors, any CloudWatch alarm action, mute rules); no recording rulesRecording and alerting rules plus Alertmanager; SNS is the only receiver
AWS vended metricsQueryable with PromQL after enabling OTel enrichmentNot native; you'd need an exporter that calls GetMetricData

Sources: OTLP endpoint limits, PromQL limits, How CloudWatch stores histogram metrics, AMP quotas, AMP Alertmanager, AMP 1.5B series and 200K rules, native histograms on AMP. For rates, see the CloudWatch pricing page and the AMP pricing page.

How each one is billed

The two backends meter different things, and classic custom metrics meter a third way. The comparison comes down to your own volume.

CloudWatch OTel metrics bill on the volume of OTLP data ingested. Each data point contributes its serialized size: the metric name, all labels, resource attributes, the timestamp, and the value. Storage for 15 months is included, console queries in Query Studio and dashboards are free, and PromQL API queries are metered per samples scanned.

AMP bills on metric samples ingested, query samples processed, and stored metrics. Ingestion is tiered by volume within a Region, so the rate drops as monthly usage climbs (understand and optimize AMP costs).

Classic custom metrics bill per metric per month, and every unique dimension combination is a separate metric. That is what makes high-cardinality dimensions expensive on the classic path.

To size the CloudWatch side, send a representative slice of your workload to the OTLP endpoint for a day and read the ingestion volume under Usage Metrics in the CloudWatch console (OTel metrics pricing and storage). Then price that measured volume on the live CloudWatch and AMP pricing pages.

Sending OpenTelemetry metrics to the CloudWatch OTLP endpoint

The getting started guide documents four options: the CloudWatch agent (an AWS-managed OpenTelemetry Collector build), an upstream OpenTelemetry Collector (contrib), a custom Collector build, or the ADOT SDK exporting directly. For production, run a collector. AWS recommends the CloudWatch agent for most customers. I show the upstream Collector because it's portable.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317   # apps can still use gRPC locally; the collector converts
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 200          # trigger: send once 200 data points are buffered (not a size cap)
    send_batch_max_size: 1000     # hard cap: split larger batches to stay within 1,000 data points per request
    timeout: 10s

exporters:
  otlphttp:
    # metrics_endpoint, not endpoint (endpoint appends /v1/metrics a second time)
    metrics_endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics
    compression: gzip             # only gzip and none are supported
    auth:
      authenticator: sigv4auth

extensions:
  sigv4auth:
    service: monitoring
    region: us-east-1

service:
  extensions: [sigv4auth]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Things that trip people up:

  • The endpoint doesn't accept gRPC, so use the otlphttp exporter, not otlp. Recent upstream releases rename it otlp_http and keep otlphttp as a deprecated alias (exporter README). AWS docs still use otlphttp.
  • The sender needs cloudwatch:PutMetricData, the same action as classic metrics (CloudWatchAgentServerPolicy on EC2; IRSA or EKS Pod Identity on EKS).
  • Outside AWS, the endpoint also accepts an API key through the bearertokenauth extension (bearer token setup), with cloudwatch:CallWithBearerToken and cloudwatch:PutMetricData (both in CloudWatchAPIKeyAccess). Keys are long-term, ingestion-only credentials: use them only where short-term IAM credentials aren't an option, set an expiration, and rotate them.
  • Over-size requests return 413; data point count, label, metadata, attribute, and timestamp violations return 400; TPS and new-series throttling return 429. A partially throttled or invalid request returns 200 with the rest ingested, so check exporter logs, not just HTTP status.
  • Drop unqueried high-cardinality labels in the collector (attributes or transform processor). Don't rely on CloudWatch Pipelines for scraped data: its destructive processors pass cumulative metrics (including scraped Prometheus counters and histograms) and AWS vended metrics through unchanged. Use metric_relabel_configs instead.

To verify, run a PromQL selector in Query Studio. Quote dotted names (OTel semantic conventions): {"http.server.request.duration"}. A bare dotted name isn't valid PromQL (PromQL querying).

CloudWatch flattens the OTLP hierarchy into labels with an @ prefix per scope:

OTLP scopeLabel formExample
Resource attribute@resource.<attr>"@resource.service.name"="checkout"
Instrumentation scope@instrumentation.@name"@instrumentation.@name"="otel-go/metrics"
Data point attributebare, or @datapoint.<attr>method="GET"
AWS-reserved@aws.<field>"@aws.region"="us-east-1"

Check histogram temporality before you write queries: CloudWatch stores both delta and cumulative OTel histograms and reports which in the __temporality__ label. rate() and increase() silently give wrong answers on delta histograms. Use sum_over_time instead, for example histogram_quantile(0.99, sum_over_time({"http.server.request.duration"}[5m])).

Scraping Prometheus exporters on EC2, ECS, and MSK without running a collector

If your metrics already come from Prometheus exporters (node, DCGM, JMX, your app's /metrics), a managed collector creates elastic network interfaces in your subnets, scrapes targets privately inside your VPC, and delivers to CloudWatch over the AWS network (managed collectors).

Prerequisites for a VPC-connected collector (setup guide):

  • A VPC with DNS enabled, and at least two subnets in different Availability Zones.
  • A dedicated collector security group that can reach your exporter ports, with those ports on the targets open only to it.
  • aps:CreateScraper and iam:CreateServiceLinkedRole (for the AWSServiceRoleForAmazonPrometheusScraper service-linked role).

Delivery needs no internet access, NAT gateway, or CloudWatch interface VPC endpoint.

Node exporter on two EC2 instances:

global:
  scrape_interval: 60s          # 30s is the minimum for managed collectors

scrape_configs:
  - job_name: ec2-node-exporter
    static_configs:
      - targets:
          - 10.0.1.10:9100
          - 10.0.1.11:9100
    metric_relabel_configs:
      # Drop series you never query before they're ingested and billed
      - source_labels: [__name__]
        regex: 'node_scrape_collector_.*'
        action: drop

Create the collector:

aws amp create-scraper \
  --alias ec2-node-exporter \
  --source '{"vpcConfiguration":{"subnetIds":["subnet-0example1","subnet-0example2"],"securityGroupIds":["sg-0example"]}}' \
  --scrape-configuration configurationBlob="$(base64 -i scrape-config.yaml)" \
  --destination '{"cloudWatchConfiguration":{"datasetArn":"arn:aws:cloudwatch:us-east-1:123456789012:dataset/default"}}'

On Linux, use base64 -w 0 instead of the macOS base64 -i. Pre-encoding works with AWS CLI v2's default cli_binary_format. On CLI v1, or with raw-in-base64-out, it double-encodes. --destination takes exactly one of ampConfiguration or cloudWatchConfiguration (create-scraper CLI reference). Check for ACTIVE with aws amp describe-scraper.

For the other sources:

  • ECS: register services in AWS Cloud Map and use dns_sd_configs with the service's DNS name, type A, and the metrics port.
  • MSK (provisioned, standard or express brokers): enable Open Monitoring (JMX exporter on port 11001, node exporter on 11002), then point dns_sd_configs at the bootstrap broker host with the b-1. prefix removed. MSK Serverless and public access combined with KRaft metadata mode aren't supported. You need one collector per cluster-and-dataset pair (MSK integration).
  • EKS uses an eksConfiguration source with Kubernetes service discovery (setup per the AMP user guide). OpenSearch uses the domain's metrics endpoint. Both, plus MSK, get automatic CloudWatch dashboards such as "EKS OTel" and "MSK OTel".

The scrape config is standard Prometheus, with limits (scraper configuration). VPC-connected collectors discover with static_configs and dns_sd_configs only. No collector supports file_sd_configs, Consul, or Eureka. remote_write and remote_read don't apply, and the config can't exceed 256 KB.

When metrics don't show up, check the collector's vended logs in CloudWatch Logs first (monitor collectors). Delivered metrics are OTel metrics, invisible to the classic metrics browser and aws cloudwatch list-metrics. Check the raw series' __type__ label in Query Studio before you port histogram queries.

AWS vended metrics in PromQL (OTel enrichment)

With OTel enrichment on, vended metrics that carry a resource identifier dimension (EC2 CPUUtilization with InstanceId, Lambda Invocations with FunctionName, and so on) become queryable in PromQL, labeled with the resource ARN, account, Region, and your resource tags. No other Prometheus backend does this. The classic metric is untouched.

Enrichment is per account, per Region, and needs resource tags on telemetry enabled first:

aws observabilityadmin start-telemetry-enrichment   # resource tags on telemetry
aws cloudwatch start-otel-enrichment                # vended metrics -> OTel/PromQL
aws cloudwatch get-otel-enrichment                  # verify: Running

CloudFormation (AWS::ObservabilityAdmin::TelemetryEnrichment, then AWS::CloudWatch::OTelEnrichment) and Terraform (aws_observabilityadmin_telemetry_enrichment, then aws_cloudwatch_otel_enrichment) work too. Deploy them in every Region you use. Then you can slice AWS metrics by tags with no exporters:

# Avg CPU of production EC2 per Application tag. CPUUtilization is a histogram;
# the scope filter excludes RDS's CPUUtilization.
avg by ("@aws.tag.Application") (
  histogram_avg(
    CPUUtilization{"@instrumentation.@name"="cloudwatch.aws/ec2", "@aws.tag.Environment"="production"}
  )
)

On histogram-typed vended metrics, min/max return an empty result, not an error (the only hint is an "ignored histogram" annotation). Use histogram_avg, histogram_quantile, histogram_count, or histogram_sum.

Enriched data points are billed as OpenTelemetry metric ingestion, and more resource tags mean more bytes per data point. Check the supported metrics list, and limit scope with --include-filters and --exclude-filters on start-otel-enrichment (a namespace plus optional metric names, up to 100 filters combined):

aws cloudwatch start-otel-enrichment \
  --include-filters '[{"Namespace":"AWS/EC2","MetricNames":["CPUUtilization"]},{"Namespace":"AWS/Lambda"}]'

Calling start-otel-enrichment again while it's running doesn't change the filters. Use UpdateOTelEnrichment.

Alarming: PromQL alarms vs. Prometheus rules

On CloudWatch, a PromQL alarm runs an instant query on an EvaluationInterval. The threshold lives in the query, and every series it returns is a separately tracked breaching contributor. PendingPeriod works like a Prometheus rule's for, RecoveryPeriod like keep_firing_for. The alarm starts in OK, not INSUFFICIENT_DATA. EvaluationPeriods and DatapointsToAlarm can't be combined with EvaluationCriteria, and TreatMissingData and EvaluationWindow don't apply (put-metric-alarm reference). A series that drops out of the result returns to OK after the recovery period.

Per-instance root disk under 10% free, on scraped node exporter metrics:

aws cloudwatch put-metric-alarm \
  --alarm-name ec2-root-disk-low \
  --evaluation-criteria '{"PromQLCriteria":{"Query":"node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"} < 0.10","PendingPeriod":300,"RecoveryPeriod":600}}' \
  --evaluation-interval 60 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts

DescribeAlarmContributors shows which instances are breaching. Alarm mute rules silence actions during maintenance. --evaluation-interval accepts 10, 20, 30, or any multiple of 60, up to 3,600 seconds. For multi-series alarms, use the CLI or API: the console's create-from-Query-Studio procedure asks for a single time series.

On AMP, you get the Prometheus model unchanged: alerting and recording rules in rule groups namespaces (2,000 per workspace by default, adjustable to 200,000) and a managed Alertmanager with grouping, inhibition, and silences (AMP Alertmanager). Amazon SNS in the same account is the only receiver. PagerDuty, Slack, and the rest go through SNS.

The gap is recording rules. CloudWatch's Prometheus-compatible APIs cover querying only: /api/v1/query, /api/v1/query_range, /api/v1/series, /api/v1/labels, and /api/v1/label/<name>/values. There is no rules or Alertmanager API. Recorded series (SLO burn-rate rules, pre-aggregated job: series) have to become direct queries within the 500-series and scan limits, or stay on AMP.

Can I point Grafana at CloudWatch with PromQL?

Yes, two ways (Grafana integration). AWS recommends the PromQL query type in the Amazon CloudWatch data source, which reuses its existing authentication and Region settings. It's in:

  • Amazon Managed Grafana workspaces on Grafana 13 or later. Grafana 13.2 workspaces became available on September 30, 2026.
  • Self-managed Grafana with CloudWatch data source plugin 12.10.0 or later, which needs Grafana ≥ 11.6.11 < 12, ≥ 12.0.10 < 12.1, ≥ 12.1.7 < 12.2, or ≥ 12.2.5.

If your Amazon Managed Grafana workspace is still on Grafana 12, or your self-managed Grafana can't run the newer plugin, use the Amazon Managed Service for Prometheus data source (plugin v3.0.0, same Grafana version ranges) with URL https://monitoring.<region>.amazonaws.com and service monitoring (sigv4Service in YAML or Terraform provisioning). The plugin defaults to aps. Leave it there and every query fails with HTTP 403 ("Credential should be scoped to correct service: 'monitoring'").

On either path, Grafana's IAM principal needs cloudwatch:GetMetricData and cloudwatch:ListMetrics. With only those, Save & test reports an error for the Logs check, but PromQL queries work.

Both paths call the PromQL HTTP API, which is metered per samples scanned, unlike console queries (OTel metrics pricing and storage). The query limits apply too. A panel with one line per pod across 2,000 pods is truncated at 500 series unless you aggregate first (sum by, topk). Aggregation doesn't help with the 7-day cap, which covers range parameters and lookback periods; a request over it fails with a 422. Split a 30-day panel into requests of 7 days or less. Grafana won't do it for you. The scan limits count underlying series regardless.

For label names, use the CloudWatch PromQL querying docs as the reference ("@aws.account", "@aws.region"). Confirm the exact names in Query Studio before you build panels on them.

Migrating custom metrics from PutMetricData/EMF to OTLP

AWS's migration guide lays out four phases, workload by workload: dual-write (OTel SDK next to your existing PutMetricData or EMF calls), validate in Query Studio, recreate alarms and dashboards on PromQL and run them side by side, then cut over. What I'd add:

  • Start with high-cardinality dimensions (per-container, per-tenant, per-endpoint), where OTLP saves the most, and keep each dual-write overlap short.
  • The ADOT CloudWatch EMF exporter (awsemf) still produces classic custom metrics; switching a collector to otlphttp with SigV4 is what changes the billing model.
  • OTel uses base units, so RequestLatency (ms) becomes http_request_duration_seconds, and the threshold moves into the PromQL query with a histogram function: histogram_avg(rate(http_request_duration_seconds{path="/api/users"}[5m])) * 1000 > 100 for cumulative (sum_over_time instead of rate for delta), passed through --evaluation-criteria like the disk alarm above. Don't copy the guide's "OTel equivalent (PromQL alarm)" example: it uses the classic --metrics/--threshold form, which can't be combined with EvaluationCriteria, and applies avg() to a histogram.
  • You don't migrate EC2 or RDS metrics. You enable enrichment.

Running both

Running both is a reasonable end state. Split by purpose rather than duplicating. For example, keep the platform team's Prometheus rules and long-standing Grafana dashboards on AMP, and send new service metrics plus AWS vended metrics to CloudWatch. For a migration window, one collector can fan out. Add these to the first config (the service block replaces the original):

extensions:
  sigv4auth/amp:
    service: aps                  # AMP signs as "aps"
    region: us-east-1

exporters:
  prometheusremotewrite/amp:
    endpoint: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-EXAMPLE/api/v1/remote_write
    auth:
      authenticator: sigv4auth/amp

service:
  extensions: [sigv4auth, sigv4auth/amp]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]         # same batch block, with send_batch_max_size: 1000
      exporters: [otlphttp, prometheusremotewrite/amp]

Current upstream builds rename prometheusremotewrite to prometheus_remote_write, keeping the old name as a deprecated alias. You pay both ingestion bills, so keep a fan-out temporary. Don't expect the two copies to match. The Prometheus Remote Write exporter:

  • Drops delta sums and delta histograms (the README also mentions summaries), so delta metrics land in CloudWatch and silently go missing in AMP. To keep them, give the AMP exporter its own pipeline with a delta-to-cumulative step, such as the contrib Delta to Cumulative processor (alpha, and stateful).
  • Translates names to Prometheus form by default (translation_strategy: UnderscoreEscapingWithSuffixes): dots become underscores, unit and _total suffixes are added, explicit-bucket histograms become _bucket/_sum/_count, and resource attributes move into target_info. CloudWatch keeps UTF-8 names and @resource.* labels, so dashboards and alerts don't port between the two as-is.

Decision matrix

Choose this......if
CloudWatch OTLP + PromQLYou want AWS vended and custom metrics in one PromQL-queryable place sliced by resource tags, CloudWatch alarms, and mostly work in the CloudWatch console
CloudWatch OTLP + PromQLYour pain is classic custom-metric cost from high-cardinality dimensions, you'd rather not run a metrics backend, or you have late or backfilled data (up to 14 days old)
CloudWatch managed collectorYou want Prometheus exporters on EC2, ECS, MSK, OpenSearch, or EKS in CloudWatch without running an agent
AMPYou have Prometheus recording rules, Alertmanager routing, or remote write from self-managed Prometheus you don't want to rework
AMPYou're at hundreds of millions of active series, or at sample rates high enough for AMP's volume-tiered ingestion to matter
AMPGrafana is your primary UI with high query volume, you need raw-series queries beyond 7 days per request, or you want native histograms end to end
AMP managed collectorYou want agentless scraping into AMP (same CreateScraper API, AMP destination)
BothYou have an existing Prometheus estate and want vended-metric PromQL and CloudWatch alarms for new services

The short version

  1. AMP or CloudWatch now that CloudWatch supports PromQL? CloudWatch for AWS and custom metrics in one place, CloudWatch alarms, and no backend to run. AMP for recording rules, Alertmanager, remote write, extreme scale, or heavy Grafana querying. Both, split by purpose, is fine.

  2. How do I send OTel metrics to the CloudWatch OTLP endpoint? A collector with otlphttp to metrics_endpoint: https://monitoring.<region>.amazonaws.com/v1/metrics, sigv4auth with service monitoring, and send_batch_max_size at or below 1,000.

  3. How do I scrape exporters on EC2/ECS/MSK without running a collector? aws amp create-scraper with a VPC source and a cloudWatchConfiguration destination.

  4. Can I point Grafana at CloudWatch with PromQL? Yes, with the PromQL query type in the Amazon CloudWatch data source, or on older Grafana, the Amazon Managed Service for Prometheus data source with service monitoring. Budget for API query charges.

  5. Is OTLP ingestion cheaper than custom metrics? The models differ in kind: OTLP on bytes ingested, classic custom metrics per metric per month, AMP on samples. High-cardinality dimensions are what make classic custom metrics expensive, and neither OTLP nor AMP charges per series. Measure your volume, then price it on the live pricing pages.

Questions welcome in the comments, especially if you've run either path on a real fleet. If you're mid-migration in production and have an AWS Support plan, open a support case.

AWS
EXPERT

published a month ago212 views