CloudWatch Now Speaks PromQL. Do You Still Need Prometheus?
Should I use AMP or CloudWatch now that CloudWatch supports PromQL? Until June the answer was short: Prometheus semantics on AWS meant Amazon Managed Service for Prometheus (AMP), and CloudWatch was where AWS service metrics lived.
This year CloudWatch started ingesting OpenTelemetry metrics over OTLP, querying them with PromQL, and, since July, scraping your exporters with managed collectors. This post covers where the two differ, how each one is billed, and how to start on either (or both), for any compute type with a /metrics endpoint.
Rates for both services change, so price your own workload on the CloudWatch pricing page and the AMP pricing page.
The four pieces
CloudWatch OpenTelemetry metrics with PromQL
Next to classic metrics (namespaces, up to 30 dimensions, PutMetricData and EMF, billed per metric per month), CloudWatch now has OpenTelemetry metrics, which AWS labels the recommended path for new workloads. It went GA in June 2026 (What's New) in all commercial Regions except Middle East (UAE), Middle East (Bahrain), and Israel (Tel Aviv). You send to a Regional OTLP/HTTP endpoint, https://monitoring.<region>.amazonaws.com/v1/metrics, and query with PromQL (Prometheus 3.0 specification, including UTF-8 names) in Query Studio or through a Prometheus-compatible HTTP API.
Supported types are gauge, sum, histogram, and exponential histogram. If your exporters expose summaries (common with JMX exporters and Java clients), check in Query Studio what actually lands.
The two models don't share query APIs. GetMetricData, ListMetrics, and Metrics Insights only see classic metrics. The PromQL API only sees OTel metrics (Metrics in Amazon CloudWatch). OTel enrichment (below) is what brings AWS vended metrics into PromQL.
Amazon Managed Service for Prometheus workspaces
A workspace is an isolated Prometheus-compatible store with its own remote write endpoint, query endpoint, ruler, and Alertmanager. You ingest with Prometheus remote write, from Prometheus in agent mode, an ADOT or OpenTelemetry Collector, or an AMP managed collector (ingest methods).
Managed collectors: one API, two destinations
The "CloudWatch managed Prometheus collector" and the "AMP managed scraper" are the same feature with the same scrape configuration and a different destination. Both are created with the CreateScraper API (aws amp create-scraper, even when the destination is CloudWatch), which sends to either an AMP workspace or a CloudWatch dataset (create-scraper CLI reference). Only the destination block differs. For AMP:
{ "ampConfiguration": { "workspaceArn": "arn:aws:aps:us-east-1:123456789012:workspace/ws-EXAMPLE" } }
For CloudWatch:
{ "cloudWatchConfiguration": { "datasetArn": "arn:aws:cloudwatch:us-east-1:123456789012:dataset/default" } }
Every account already has a default CloudWatch dataset in each Region. The CloudWatch destination launched on July 31, 2026 and is available wherever the CloudWatch OTLP endpoint is, except Asia Pacific (New Zealand), ap-southeast-6 (listed as "Asia Pacific (Auckland)" in the PromQL Regions table).
Either flavor is charged by the hour on top of the ingestion charge at its destination. Rates are on the CloudWatch and AMP pricing pages.
Side by side: where they differ
| CloudWatch OTel metrics | Amazon Managed Service for Prometheus | |
|---|---|---|
| Ingestion protocol | OTLP over HTTP 1.1 only (no gRPC) | Prometheus remote write |
| Ingestion metering | Bytes of uncompressed OTLP payload ingested | Metric samples ingested, tiered by volume within a Region |
| Storage | 15 months included | Billed separately; 150 days default retention, up to 1,095 days |
| Query metering | Console (Query Studio, dashboards): free. PromQL API: samples scanned | Query samples processed |
| Scale | Per account: 500 requests/s; 1,000 data points and 5 MB uncompressed per request; 1M new series per 10 minutes | 50M active series default, up to 1.5B per workspace; ~1.67M samples/s per workspace (adjustable) |
| Late data | Timestamps up to 14 days in the past, 2 hours in the future | Samples older than 1 hour refused; out-of-order window 60 s default, up to 600 s |
| Query limits | 500 series returned per query; 7-day max range per request; 100,000 series and 300M samples scanned per 24 h window | 95-day max range; 12M series fetched and 50M samples per 24-hour interval of a single query |
| Histograms | All histograms stored as exponential histograms (explicit-bucket histograms are converted at ingestion); delta and cumulative | Classic histograms plus Prometheus native histograms (GA June 2026) |
| Rules and alerting | PromQL alarms (per-series contributors, any CloudWatch alarm action, mute rules); no recording rules | Recording and alerting rules plus Alertmanager; SNS is the only receiver |
| AWS vended metrics | Queryable with PromQL after enabling OTel enrichment | Not native; you'd need an exporter that calls GetMetricData |
Sources: OTLP endpoint limits, PromQL limits, How CloudWatch stores histogram metrics, AMP quotas, AMP Alertmanager, AMP 1.5B series and 200K rules, native histograms on AMP. For rates, see the CloudWatch pricing page and the AMP pricing page.
How each one is billed
The two backends meter different things, and classic custom metrics meter a third way. The comparison comes down to your own volume.
CloudWatch OTel metrics bill on the volume of OTLP data ingested. Each data point contributes its serialized size: the metric name, all labels, resource attributes, the timestamp, and the value. Storage for 15 months is included, console queries in Query Studio and dashboards are free, and PromQL API queries are metered per samples scanned.
AMP bills on metric samples ingested, query samples processed, and stored metrics. Ingestion is tiered by volume within a Region, so the rate drops as monthly usage climbs (understand and optimize AMP costs).
Classic custom metrics bill per metric per month, and every unique dimension combination is a separate metric. That is what makes high-cardinality dimensions expensive on the classic path.
To size the CloudWatch side, send a representative slice of your workload to the OTLP endpoint for a day and read the ingestion volume under Usage Metrics in the CloudWatch console (OTel metrics pricing and storage). Then price that measured volume on the live CloudWatch and AMP pricing pages.
Sending OpenTelemetry metrics to the CloudWatch OTLP endpoint
The getting started guide documents four options: the CloudWatch agent (an AWS-managed OpenTelemetry Collector build), an upstream OpenTelemetry Collector (contrib), a custom Collector build, or the ADOT SDK exporting directly. For production, run a collector. AWS recommends the CloudWatch agent for most customers. I show the upstream Collector because it's portable.
receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 # apps can still use gRPC locally; the collector converts http: endpoint: 0.0.0.0:4318 processors: batch: send_batch_size: 200 # trigger: send once 200 data points are buffered (not a size cap) send_batch_max_size: 1000 # hard cap: split larger batches to stay within 1,000 data points per request timeout: 10s exporters: otlphttp: # metrics_endpoint, not endpoint (endpoint appends /v1/metrics a second time) metrics_endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics compression: gzip # only gzip and none are supported auth: authenticator: sigv4auth extensions: sigv4auth: service: monitoring region: us-east-1 service: extensions: [sigv4auth] pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [otlphttp]
Things that trip people up:
- The endpoint doesn't accept gRPC, so use the
otlphttpexporter, nototlp. Recent upstream releases rename itotlp_httpand keepotlphttpas a deprecated alias (exporter README). AWS docs still useotlphttp. - The sender needs
cloudwatch:PutMetricData, the same action as classic metrics (CloudWatchAgentServerPolicyon EC2; IRSA or EKS Pod Identity on EKS). - Outside AWS, the endpoint also accepts an API key through the
bearertokenauthextension (bearer token setup), withcloudwatch:CallWithBearerTokenandcloudwatch:PutMetricData(both inCloudWatchAPIKeyAccess). Keys are long-term, ingestion-only credentials: use them only where short-term IAM credentials aren't an option, set an expiration, and rotate them. - Over-size requests return 413; data point count, label, metadata, attribute, and timestamp violations return 400; TPS and new-series throttling return 429. A partially throttled or invalid request returns 200 with the rest ingested, so check exporter logs, not just HTTP status.
- Drop unqueried high-cardinality labels in the collector (
attributesortransformprocessor). Don't rely on CloudWatch Pipelines for scraped data: its destructive processors pass cumulative metrics (including scraped Prometheus counters and histograms) and AWS vended metrics through unchanged. Usemetric_relabel_configsinstead.
To verify, run a PromQL selector in Query Studio. Quote dotted names (OTel semantic conventions): {"http.server.request.duration"}. A bare dotted name isn't valid PromQL (PromQL querying).
CloudWatch flattens the OTLP hierarchy into labels with an @ prefix per scope:
| OTLP scope | Label form | Example |
|---|---|---|
| Resource attribute | @resource.<attr> | "@resource.service.name"="checkout" |
| Instrumentation scope | @instrumentation.@name | "@instrumentation.@name"="otel-go/metrics" |
| Data point attribute | bare, or @datapoint.<attr> | method="GET" |
| AWS-reserved | @aws.<field> | "@aws.region"="us-east-1" |
Check histogram temporality before you write queries: CloudWatch stores both delta and cumulative OTel histograms and reports which in the __temporality__ label. rate() and increase() silently give wrong answers on delta histograms. Use sum_over_time instead, for example histogram_quantile(0.99, sum_over_time({"http.server.request.duration"}[5m])).
Scraping Prometheus exporters on EC2, ECS, and MSK without running a collector
If your metrics already come from Prometheus exporters (node, DCGM, JMX, your app's /metrics), a managed collector creates elastic network interfaces in your subnets, scrapes targets privately inside your VPC, and delivers to CloudWatch over the AWS network (managed collectors).
Prerequisites for a VPC-connected collector (setup guide):
- A VPC with DNS enabled, and at least two subnets in different Availability Zones.
- A dedicated collector security group that can reach your exporter ports, with those ports on the targets open only to it.
aps:CreateScraperandiam:CreateServiceLinkedRole(for theAWSServiceRoleForAmazonPrometheusScraperservice-linked role).
Delivery needs no internet access, NAT gateway, or CloudWatch interface VPC endpoint.
Node exporter on two EC2 instances:
global: scrape_interval: 60s # 30s is the minimum for managed collectors scrape_configs: - job_name: ec2-node-exporter static_configs: - targets: - 10.0.1.10:9100 - 10.0.1.11:9100 metric_relabel_configs: # Drop series you never query before they're ingested and billed - source_labels: [__name__] regex: 'node_scrape_collector_.*' action: drop
Create the collector:
aws amp create-scraper \ --alias ec2-node-exporter \ --source '{"vpcConfiguration":{"subnetIds":["subnet-0example1","subnet-0example2"],"securityGroupIds":["sg-0example"]}}' \ --scrape-configuration configurationBlob="$(base64 -i scrape-config.yaml)" \ --destination '{"cloudWatchConfiguration":{"datasetArn":"arn:aws:cloudwatch:us-east-1:123456789012:dataset/default"}}'
On Linux, use base64 -w 0 instead of the macOS base64 -i. Pre-encoding works with AWS CLI v2's default cli_binary_format. On CLI v1, or with raw-in-base64-out, it double-encodes. --destination takes exactly one of ampConfiguration or cloudWatchConfiguration (create-scraper CLI reference). Check for ACTIVE with aws amp describe-scraper.
For the other sources:
- ECS: register services in AWS Cloud Map and use
dns_sd_configswith the service's DNS name, typeA, and the metrics port. - MSK (provisioned, standard or express brokers): enable Open Monitoring (JMX exporter on port 11001, node exporter on 11002), then point
dns_sd_configsat the bootstrap broker host with theb-1.prefix removed. MSK Serverless and public access combined with KRaft metadata mode aren't supported. You need one collector per cluster-and-dataset pair (MSK integration). - EKS uses an
eksConfigurationsource with Kubernetes service discovery (setup per the AMP user guide). OpenSearch uses the domain's metrics endpoint. Both, plus MSK, get automatic CloudWatch dashboards such as "EKS OTel" and "MSK OTel".
The scrape config is standard Prometheus, with limits (scraper configuration). VPC-connected collectors discover with static_configs and dns_sd_configs only. No collector supports file_sd_configs, Consul, or Eureka. remote_write and remote_read don't apply, and the config can't exceed 256 KB.
When metrics don't show up, check the collector's vended logs in CloudWatch Logs first (monitor collectors). Delivered metrics are OTel metrics, invisible to the classic metrics browser and aws cloudwatch list-metrics. Check the raw series' __type__ label in Query Studio before you port histogram queries.
AWS vended metrics in PromQL (OTel enrichment)
With OTel enrichment on, vended metrics that carry a resource identifier dimension (EC2 CPUUtilization with InstanceId, Lambda Invocations with FunctionName, and so on) become queryable in PromQL, labeled with the resource ARN, account, Region, and your resource tags. No other Prometheus backend does this. The classic metric is untouched.
Enrichment is per account, per Region, and needs resource tags on telemetry enabled first:
aws observabilityadmin start-telemetry-enrichment # resource tags on telemetry aws cloudwatch start-otel-enrichment # vended metrics -> OTel/PromQL aws cloudwatch get-otel-enrichment # verify: Running
CloudFormation (AWS::ObservabilityAdmin::TelemetryEnrichment, then AWS::CloudWatch::OTelEnrichment) and Terraform (aws_observabilityadmin_telemetry_enrichment, then aws_cloudwatch_otel_enrichment) work too. Deploy them in every Region you use. Then you can slice AWS metrics by tags with no exporters:
# Avg CPU of production EC2 per Application tag. CPUUtilization is a histogram; # the scope filter excludes RDS's CPUUtilization. avg by ("@aws.tag.Application") ( histogram_avg( CPUUtilization{"@instrumentation.@name"="cloudwatch.aws/ec2", "@aws.tag.Environment"="production"} ) )
On histogram-typed vended metrics, min/max return an empty result, not an error (the only hint is an "ignored histogram" annotation). Use histogram_avg, histogram_quantile, histogram_count, or histogram_sum.
Enriched data points are billed as OpenTelemetry metric ingestion, and more resource tags mean more bytes per data point. Check the supported metrics list, and limit scope with --include-filters and --exclude-filters on start-otel-enrichment (a namespace plus optional metric names, up to 100 filters combined):
aws cloudwatch start-otel-enrichment \ --include-filters '[{"Namespace":"AWS/EC2","MetricNames":["CPUUtilization"]},{"Namespace":"AWS/Lambda"}]'
Calling start-otel-enrichment again while it's running doesn't change the filters. Use UpdateOTelEnrichment.
Alarming: PromQL alarms vs. Prometheus rules
On CloudWatch, a PromQL alarm runs an instant query on an EvaluationInterval. The threshold lives in the query, and every series it returns is a separately tracked breaching contributor. PendingPeriod works like a Prometheus rule's for, RecoveryPeriod like keep_firing_for. The alarm starts in OK, not INSUFFICIENT_DATA. EvaluationPeriods and DatapointsToAlarm can't be combined with EvaluationCriteria, and TreatMissingData and EvaluationWindow don't apply (put-metric-alarm reference). A series that drops out of the result returns to OK after the recovery period.
Per-instance root disk under 10% free, on scraped node exporter metrics:
aws cloudwatch put-metric-alarm \ --alarm-name ec2-root-disk-low \ --evaluation-criteria '{"PromQLCriteria":{"Query":"node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"} < 0.10","PendingPeriod":300,"RecoveryPeriod":600}}' \ --evaluation-interval 60 \ --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts
DescribeAlarmContributors shows which instances are breaching. Alarm mute rules silence actions during maintenance. --evaluation-interval accepts 10, 20, 30, or any multiple of 60, up to 3,600 seconds. For multi-series alarms, use the CLI or API: the console's create-from-Query-Studio procedure asks for a single time series.
On AMP, you get the Prometheus model unchanged: alerting and recording rules in rule groups namespaces (2,000 per workspace by default, adjustable to 200,000) and a managed Alertmanager with grouping, inhibition, and silences (AMP Alertmanager). Amazon SNS in the same account is the only receiver. PagerDuty, Slack, and the rest go through SNS.
The gap is recording rules. CloudWatch's Prometheus-compatible APIs cover querying only: /api/v1/query, /api/v1/query_range, /api/v1/series, /api/v1/labels, and /api/v1/label/<name>/values. There is no rules or Alertmanager API. Recorded series (SLO burn-rate rules, pre-aggregated job: series) have to become direct queries within the 500-series and scan limits, or stay on AMP.
Can I point Grafana at CloudWatch with PromQL?
Yes, two ways (Grafana integration). AWS recommends the PromQL query type in the Amazon CloudWatch data source, which reuses its existing authentication and Region settings. It's in:
- Amazon Managed Grafana workspaces on Grafana 13 or later. Grafana 13.2 workspaces became available on September 30, 2026.
- Self-managed Grafana with CloudWatch data source plugin 12.10.0 or later, which needs Grafana ≥ 11.6.11 < 12, ≥ 12.0.10 < 12.1, ≥ 12.1.7 < 12.2, or ≥ 12.2.5.
If your Amazon Managed Grafana workspace is still on Grafana 12, or your self-managed Grafana can't run the newer plugin, use the Amazon Managed Service for Prometheus data source (plugin v3.0.0, same Grafana version ranges) with URL https://monitoring.<region>.amazonaws.com and service monitoring (sigv4Service in YAML or Terraform provisioning). The plugin defaults to aps. Leave it there and every query fails with HTTP 403 ("Credential should be scoped to correct service: 'monitoring'").
On either path, Grafana's IAM principal needs cloudwatch:GetMetricData and cloudwatch:ListMetrics. With only those, Save & test reports an error for the Logs check, but PromQL queries work.
Both paths call the PromQL HTTP API, which is metered per samples scanned, unlike console queries (OTel metrics pricing and storage). The query limits apply too. A panel with one line per pod across 2,000 pods is truncated at 500 series unless you aggregate first (sum by, topk). Aggregation doesn't help with the 7-day cap, which covers range parameters and lookback periods; a request over it fails with a 422. Split a 30-day panel into requests of 7 days or less. Grafana won't do it for you. The scan limits count underlying series regardless.
For label names, use the CloudWatch PromQL querying docs as the reference ("@aws.account", "@aws.region"). Confirm the exact names in Query Studio before you build panels on them.
Migrating custom metrics from PutMetricData/EMF to OTLP
AWS's migration guide lays out four phases, workload by workload: dual-write (OTel SDK next to your existing PutMetricData or EMF calls), validate in Query Studio, recreate alarms and dashboards on PromQL and run them side by side, then cut over. What I'd add:
- Start with high-cardinality dimensions (per-container, per-tenant, per-endpoint), where OTLP saves the most, and keep each dual-write overlap short.
- The ADOT CloudWatch EMF exporter (
awsemf) still produces classic custom metrics; switching a collector tootlphttpwith SigV4 is what changes the billing model. - OTel uses base units, so
RequestLatency (ms)becomeshttp_request_duration_seconds, and the threshold moves into the PromQL query with a histogram function:histogram_avg(rate(http_request_duration_seconds{path="/api/users"}[5m])) * 1000 > 100for cumulative (sum_over_timeinstead ofratefor delta), passed through--evaluation-criterialike the disk alarm above. Don't copy the guide's "OTel equivalent (PromQL alarm)" example: it uses the classic--metrics/--thresholdform, which can't be combined withEvaluationCriteria, and appliesavg()to a histogram. - You don't migrate EC2 or RDS metrics. You enable enrichment.
Running both
Running both is a reasonable end state. Split by purpose rather than duplicating. For example, keep the platform team's Prometheus rules and long-standing Grafana dashboards on AMP, and send new service metrics plus AWS vended metrics to CloudWatch. For a migration window, one collector can fan out. Add these to the first config (the service block replaces the original):
extensions: sigv4auth/amp: service: aps # AMP signs as "aps" region: us-east-1 exporters: prometheusremotewrite/amp: endpoint: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-EXAMPLE/api/v1/remote_write auth: authenticator: sigv4auth/amp service: extensions: [sigv4auth, sigv4auth/amp] pipelines: metrics: receivers: [otlp] processors: [batch] # same batch block, with send_batch_max_size: 1000 exporters: [otlphttp, prometheusremotewrite/amp]
Current upstream builds rename prometheusremotewrite to prometheus_remote_write, keeping the old name as a deprecated alias. You pay both ingestion bills, so keep a fan-out temporary. Don't expect the two copies to match. The Prometheus Remote Write exporter:
- Drops delta sums and delta histograms (the README also mentions summaries), so delta metrics land in CloudWatch and silently go missing in AMP. To keep them, give the AMP exporter its own pipeline with a delta-to-cumulative step, such as the contrib Delta to Cumulative processor (alpha, and stateful).
- Translates names to Prometheus form by default (
translation_strategy: UnderscoreEscapingWithSuffixes): dots become underscores, unit and_totalsuffixes are added, explicit-bucket histograms become_bucket/_sum/_count, and resource attributes move intotarget_info. CloudWatch keeps UTF-8 names and@resource.*labels, so dashboards and alerts don't port between the two as-is.
Decision matrix
| Choose this... | ...if |
|---|---|
| CloudWatch OTLP + PromQL | You want AWS vended and custom metrics in one PromQL-queryable place sliced by resource tags, CloudWatch alarms, and mostly work in the CloudWatch console |
| CloudWatch OTLP + PromQL | Your pain is classic custom-metric cost from high-cardinality dimensions, you'd rather not run a metrics backend, or you have late or backfilled data (up to 14 days old) |
| CloudWatch managed collector | You want Prometheus exporters on EC2, ECS, MSK, OpenSearch, or EKS in CloudWatch without running an agent |
| AMP | You have Prometheus recording rules, Alertmanager routing, or remote write from self-managed Prometheus you don't want to rework |
| AMP | You're at hundreds of millions of active series, or at sample rates high enough for AMP's volume-tiered ingestion to matter |
| AMP | Grafana is your primary UI with high query volume, you need raw-series queries beyond 7 days per request, or you want native histograms end to end |
| AMP managed collector | You want agentless scraping into AMP (same CreateScraper API, AMP destination) |
| Both | You have an existing Prometheus estate and want vended-metric PromQL and CloudWatch alarms for new services |
The short version
-
AMP or CloudWatch now that CloudWatch supports PromQL? CloudWatch for AWS and custom metrics in one place, CloudWatch alarms, and no backend to run. AMP for recording rules, Alertmanager, remote write, extreme scale, or heavy Grafana querying. Both, split by purpose, is fine.
-
How do I send OTel metrics to the CloudWatch OTLP endpoint? A collector with
otlphttptometrics_endpoint: https://monitoring.<region>.amazonaws.com/v1/metrics,sigv4authwith servicemonitoring, andsend_batch_max_sizeat or below 1,000. -
How do I scrape exporters on EC2/ECS/MSK without running a collector?
aws amp create-scraperwith a VPC source and acloudWatchConfigurationdestination. -
Can I point Grafana at CloudWatch with PromQL? Yes, with the PromQL query type in the Amazon CloudWatch data source, or on older Grafana, the Amazon Managed Service for Prometheus data source with service
monitoring. Budget for API query charges. -
Is OTLP ingestion cheaper than custom metrics? The models differ in kind: OTLP on bytes ingested, classic custom metrics per metric per month, AMP on samples. High-cardinality dimensions are what make classic custom metrics expensive, and neither OTLP nor AMP charges per series. Measure your volume, then price it on the live pricing pages.
Questions welcome in the comments, especially if you've run either path on a real fleet. If you're mid-migration in production and have an AWS Support plan, open a support case.
- Topics
- Management & Governance
- Language
- English
Relevant content
asked 2 years ago
- Accepted Answer
asked 3 years ago
- Accepted Answer
asked 2 years ago
AWS OFFICIALUpdated 8 months ago
AWS OFFICIALUpdated 2 years ago
AWS OFFICIALUpdated 3 years ago