How do I troubleshoot instance metadata issues on my EC2 Linux instance?

Short description

Instance metadata is information about the EC2 instance. Instance metadata includes the instance ID, public and private IP addresses, security groups, AWS Identity and Access Management (IAM) roles, and other information. The instance metadata is exposed to the instance through HTTP requests to the IP address 169.254.169.254.

You might receive the following issues when you retrieve instance metadata from the Instance Metadata Service (IMDS):

• HTTP request errors
• Proxy configuration
• Local firewall rules
• Request throttling

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

HTTP request errors

When you make an HTTP request to retrieve instance metadata, the following HTTP error codes might be returned:

404 - Not Found

This error applies to IMDS v1. You get this error when the resource that you requested isn't available. Verify that you're using the correct URL.

400 - Bad Request

This error applies to IMDS v2. You get this error when the PUT request isn't valid.

401 - Unauthorized

You get this error when the GET request uses a token that isn't valid. To resolve this error, generate a new token.

403 - Forbidden

You get this error when the request isn't allowed or when IMDS isn't activated. To check IMDS status, run the following describe-instances AWS CLI command:

$ aws ec2 describe-instances --instance-ids your_instance_id --query 'Reservations[].Instances[].MetadataOptions'

Note: In the preceding command, replace your_instance_id with your actual instance ID.

The proceeding command produces output similar to the following:

[
  {
    "State": "applied",
    "HttpTokens": "optional",
    "HttpPutResponseHopLimit": 1,
    "HttpEndpoint": "disabled",
    "HttpProtocolIpv6": "disabled",
    "InstanceMetadataTags": "disabled"
  }
]

If the HttpEndpoint value in the output is disabled, then run the following modify-instance-metadata-options command:

$ aws ec2 modify-instance-metadata-options \
    --instance-id your_instance_id \
    --http-endpoint enabled

Proxy configuration

If you're using a proxy to access the internet, then you must exclude the IMDS IP address (169.254.169.254). If you don't exclude the IMDS IP address, then you might not be able to retrieve instance metadata.

To exclude the IMDS IP address from proxy use, set a no_proxy environment variable with the following address:

$ export no_proxy='169.254.169.254,[fd00:ec2::254]'

Local firewall rules

Firewalls in the instance might prevent access to the IMDS by some processes.

Make sure that firewalls in the instance's operating system don't block outgoing traffic to the instance metadata IP address. Firewalls include iptables, UFW (uncomplicated firewall), and so on.

To check firewall rules for iptables, run the following command:

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REJECT tcp -- anywhere 169.254.169.254 owner UID match 1000-10000 reject-with icmp-port-unreachable

If the output from iptables shows a metadata retrieval rejection, then you might receive the following error:

$ curl http://169.254.169.254/latest/meta-data/
curl: (7) Failed to connect to 169.254.169.254 port 80 after 0 ms: Connection refused

To resolve this error, run the following command to delete the rule:

$ sudo iptables -D OUTPUT -proto tcp -destination 169.254.169.254 -match owner -uid-owner 1000-10000 -jump REJECT

Request throttling

Traffic to the IMDS is throttled based on the number of packets per second. There's a limit of 1024 PPS for each Elastic Network Interface attached to the instance. If your PPS rate to the IMDS service exceeds 1024 PPS, then the request is throttled.

If your requests are throttled, then retry your request with an exponential backoff strategy.

To view how often your EC2 instances reach their throttling limits, check the linklocal_allowance_exceeded metric in the Elastic Network Interface driver. This metric shows that the number of packets shaped by the traffic to local proxy services exceeds the maximum for the network interface.

To view the linklocal_allowance_exceeded metric, run the following command to:

$ ethtool -S eth0

Note: In the preceding command, replace eth0 with the name of your network interface.

Related information

Using a proxy on Amazon EC2 instances

Access instance metadata for an EC2 instance

Limit access to the Instance Metadata Service

Query throttling

Metrics for the ENA driver