How do I troubleshoot 504 Gateway Timeout errors on an Amazon CloudWatch data source when my Amazon Managed Grafana workspace has private VPC connectivity to CloudWatch?

Issue

I have an Amazon Managed Grafana (AMG) workspace with VPC connection enabled, the workspace's network access points to a VPC in my account, not "Public access". When I click Save & Test on an Amazon CloudWatch data source, the request returns 504 Gateway Timeout.

Short description

This article covers only the VPC-connected topology. If your workspace uses Public access, Grafana reaches the CloudWatch API over the public internet and the diagnosis is different.

The CloudWatch data source talks to two service endpoints. Save & Test can fail for one and pass for the other, so treat them as independent paths:

API used by the data source	Public DNS name	Interface VPC endpoint service name
CloudWatch Metrics	monitoring.<region>.amazonaws.com	com.amazonaws.<region>.monitoring
CloudWatch Logs	logs.<region>.amazonaws.com	com.amazonaws.<region>.logs

Not the same endpoint: com.amazonaws.<region>.grafana-workspace carries the Grafana workspace's own control-plane traffic. It is not what the CloudWatch data source uses. Do not confuse it with the two endpoints above.

A 504 on Save & Test almost always means the workspace ENIs can't complete a TCP 443 connection to one of those two endpoints. The rest of this article finds out why. This article addresses the common topology patterns and the steps to troubleshoot and resolve this issue.

What a healthy call looks like

Before chasing a 504, know what "good" looks like. Connectivity is healthy when all three of these hold, in order, from an EC2 instance in the workspace subnet:

1. DNS resolves to a private IP — dig returns an address in your VPC's (or the networking VPC's) CIDR, not a public IP.
2. TCP 443 completes — the connection establishes instead of hanging.
3. TLS verifies — the presented certificate's subject is CN=monitoring.<region>.amazonaws.com.

If all three pass, the workspace can reach the endpoint.

A failing call, by contrast, hangs on TCP connect after DNS resolves, step 1 passes but step 2 never does:

$ curl -v https://monitoring.<region>.amazonaws.com
*   Trying 10.x.x.x:443...
(hangs, eventually times out)

The 404 UnknownOperationException is also healthy. Once TLS completes, an unsigned GET / reaches the CloudWatch frontend, which rejects it with HTTP/1.1 404 and an UnknownOperationException body. That 404 is not an error in the connectivity path, it's proof you reached the service. Only a signed API call (which the Grafana data source makes) would return data.

[ec2-user@ip-10-0-36-71 ~]$ dig +short monitoring.eu-west-1.amazonaws.com
10.0.26.247
[ec2-user@ip-10-0-36-71 ~]$ curl -v https://monitoring.eu-west-1.amazonaws.com
*   Trying 10.0.26.247:443...
* Connected to monitoring.eu-west-1.amazonaws.com (10.0.26.247) port 443
* SSL certificate verified via OpenSSL.
*   subject: CN=monitoring.eu-west-1.amazonaws.com
< HTTP/1.1 404 Not Found
< x-amzn-RequestId: 73db7eb4-aafb-4ae9-a7de-0a9591df0a69
<UnknownOperationException/>

Resolution

Step 1 : Run the fast-path triage

From an EC2 instance in one of the workspace's subnets, run:

REGION=<region>
dig +short monitoring.$REGION.amazonaws.com
timeout 5 bash -c "</dev/tcp/monitoring.$REGION.amazonaws.com/443" && echo "TCP OK"

Match the result and jump to the right section:

dig returns	TCP 443	Diagnosis	Go to
Private IP inside the workspace VPC CIDR	opens	Metrics path is healthy — check the Logs endpoint the same way	See Verifying the fix
Private IP inside the workspace VPC CIDR	hangs	Endpoint reachable by DNS but not by packets	See Scenario 1
Private IP in another VPC's CIDR	hangs	Cross-VPC routing or remote SG	See Scenario 2
dig times out (resolver error)	—	A Resolver forwarding rule is hijacking the query	See Scenario 2
A public IP	—	Private connectivity isn't in effect — outside this article's scope	—

---

Scenario 1 — single-VPC

The interface VPC endpoint lives in the workspace VPC. dig returns a private IP inside the workspace VPC CIDR, and TCP 443 hangs.

Reference setup

• AMG workspace connected to vpc-A (subnets in two AZs), single account.
• com.amazonaws.<region>.monitoring and .logs endpoints in vpc-A, Private DNS enabled, with ENIs in the same AZs as the workspace subnets.
• vpc-A has enableDnsSupport=true and enableDnsHostnames=true (both required for Private DNS).
• Endpoint SG allows inbound TCP 443 from the workspace SG (or vpc-A's CIDR); workspace SG allows outbound TCP 443.

Find the cause

What you observe	Cause	Fix
dig OK, TCP 443 hangs, endpoint is available	Endpoint SG has no inbound TCP 443 from the workspace SG/CIDR (DNS works independently of packet forwarding)	Add inbound TCP 443 from the workspace SG or vpc-A CIDR
dig OK, TCP 443 hangs, endpoint SG looks fine	Custom NACL on the workspace subnet blocks return traffic (NACLs are stateless — an open SG won't compensate)	Allow inbound ephemeral ports 1024–65535 and outbound TCP 443
Works in one AZ, hangs in another	Endpoint has no ENI in the workspace subnet's AZ	Add an endpoint subnet/ENI in every AZ the workspace uses
Metrics passes, Logs times out	com.amazonaws.<region>.logs endpoint or its DNS is missing	Create the logs endpoint (with Private DNS)

Confirm the config while diagnosing

aws grafana describe-workspace --workspace-id g-xyz          # subnetIds + securityGroupIds populated
aws ec2 describe-vpc-attribute --vpc-id vpc-A --attribute enableDnsSupport
aws ec2 describe-vpc-attribute --vpc-id vpc-A --attribute enableDnsHostnames
aws ec2 describe-vpc-endpoints \
  --filters Name=vpc-id,Values=vpc-A \
    Name=service-name,Values=com.amazonaws.<region>.monitoring,com.amazonaws.<region>.logs

Both endpoints must be State: available, PrivateDnsEnabled=true, and cover the workspace AZs.

---

Scenario 2 — cross-VPC

The endpoint lives in a central networking VPC/account. dig returns a private IP outside the workspace VPC CIDR (or times out), and TCP 443 hangs.

Cross-VPC is Scenario 1 plus two extra concerns: (a) DNS has to point at the remote endpoint, and (b) the workspace subnet needs a route that actually reaches it.

Reference setup

• AMG workspace connected to vpc-A in workspace account.
• com.amazonaws.<region>.monitoring (and .logs) endpoints in vpc-N, networking account, Private DNS disabled.
• DNS delivery — one of:
  • B.1 — a Private Hosted Zone monitoring.<region>.amazonaws.com in networking account, apex ALIAS → the endpoint's regional DNS name, cross-account associated with vpc-A.
  • B.2 — a PHZ in the networking account associated with vpc-N, plus a Route 53 Resolver FORWARD rule for the CloudWatch domain targeting an inbound resolver endpoint in vpc-N, RAM-shared to workspace account and associated with vpc-A.
• Routing: workspace subnet route tables have an explicit route to vpc-N's CIDR via Transit Gateway or VPC peering.
• Endpoint SG in the networking account allows inbound TCP 443 from vpc-A's CIDR — not the workspace SG, because SG references don't cross accounts.

Find the cause

What you observe	Cause	Fix
dig returns IP in vpc-N, TCP 443 hangs; workspace subnet has 0.0.0.0/0 → NAT but no route to vpc-N	No route to vpc-N's CIDR. A NAT gateway only carries traffic to public destinations; it never reaches a private IP in another VPC, regardless of the default route	Add an explicit route to vpc-N's CIDR via TGW or peering
dig returns a public IP (B.1)	Cross-account PHZ association not completed	Finish the two-step association
dig never forwards / wrong answer (B.2)	RAM-shared FORWARD rule not COMPLETE	Re-share/associate the rule; confirm Status: COMPLETE
dig OK to vpc-N IP, route exists, TCP 443 still hangs	Endpoint SG in the networking account doesn't allow vpc-A's CIDR	Add inbound TCP 443 for vpc-A's CIDR (or a customer-managed prefix list)
dig times out with a resolver error	A stray FORWARD rule captures the CloudWatch domain and points at an unreachable target	Correct the rule's target to a valid inbound resolver endpoint
Metrics passes, Logs times out	logs endpoint or its DNS record/rule is missing	Add the logs endpoint and a matching PHZ record or Resolver rule

Confirm DNS delivery (from the workspace account)

# Pattern B.1 — the PHZ should be associated with vpc-A
aws route53 list-hosted-zones-by-vpc --vpc-id vpc-A --vpc-region <region>

# Pattern B.2 — the FORWARD rule should show Status: COMPLETE
aws route53resolver list-resolver-rule-associations --filters Name=VPCId,Values=vpc-A

# The workspace subnet needs an active route to vpc-N's CIDR via TGW/peering
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<workspace-subnet-id>

No access to the networking account? The workspace-side signals (dig, the TCP test, curl) still distinguish a DNS failure from a routing failure from an SG failure — enough to hand the right fix to the networking team.

---

Verifying the fix

After each change, click Save & Test. Both Metrics and Logs should pass; if only one does, revisit the endpoint or DNS record for the failing service.

From inside the workspace VPC, the healthy end state — run for both monitoring and logs:

REGION=<region>
for host in monitoring.$REGION.amazonaws.com logs.$REGION.amazonaws.com; do
  echo "== $host =="
  dig +short "$host"                                              # a private IP
  timeout 5 bash -c "</dev/tcp/$host/443" && echo "TCP OK"        # TCP 443 opens
  echo | openssl s_client -connect "$host:443" -servername "$host" 2>/dev/null \
    | openssl x509 -noout -subject                                # subject=CN = <host>
  curl -sv "https://$host" 2>&1 | grep -E 'HTTP/|UnknownOperationException'
done
# Expected: HTTP/1.1 404 Not Found  +  <UnknownOperationException/>

The Amazon VPC Reachability Analyzer can validate the path end-to-end from a workspace ENI to the endpoint ENI.

Related documentation

• Connect to data sources or notification channels in Amazon VPC from Amazon Managed Grafana
• Using CloudWatch, CloudWatch Synthetics, and CloudWatch Network Monitoring with interface VPC endpoints
• Why can't I resolve my DNS query using Route 53 Resolver rules and endpoints
• Private DNS for interface endpoints
• DNS attributes for your VPC
• Associating more than one VPC with a private hosted zone
• Centralized DNS management of hybrid cloud with Amazon Route 53 and AWS Transit Gateway
• Amazon VPC Reachability Analyzer