Cannot stop EC2 instance - InternalError

0

Since July 29, we have discovered that we are unable to stop at least two EC2 instances. These instances are i-eb4f3b88 and i-af40dccc. They are both located in the us-east-1b availability zone, but many other instances we have in the same availability zone are not affected by this issue. When we try to do so, we get an "internal error":

Failed to stop the instance i-eb4f3b88 - An internal error has occurred

Using the SDK and CLI results in similar error messages. I can confirm I have the IAM permissions necessary to call the StopInstances API on the two instances mentioned. I am aware that there was increased error rates for EC2 APIs reported between 1:03PM and 1:49PM PDT, but this issue has been affecting us long before then and it is still affecting us as of 3:50PM PDT.

Could someone please look at it from the AWS side, to see if we are impacted by a strange issue?

The SDK error message is "message=An internal error has occurred, code=InternalError, time=Thu Aug 03 2023 10:07:55 GMT+0000 (Coordinated Universal Time), requestId=13a56eb6-c9e1-4a80-8c9c-26bfeb472bc9, statusCode=500, retryable=true"

Bottom part of the CLI error message (after running aws --debug ec2 stop-instances --instance-ids i-eb4f3b88) is:

2023-08-03 20:05:53,377 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ec2.us-east-1.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.13.3 Python/3.11.4 Linux/5.19.0-50-generic exe/x86_64.ubuntu.22 prompt/off command/ec2.stop-instances', 'X-Amz-Date': b'20230803T223553Z', 'X-Amz-Security-Token': b'redacted security token', 'Authorization': b'AWS4-HMAC-SHA256 Credential=redacted role name/20230803/us-east-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=redacted signature', 'Content-Length': '63'}>

2023-08-03 20:05:53,377 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.13.3/dist/awscli/botocore/cacert.pem

2023-08-03 20:05:53,377 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: ec2.us-east-1.amazonaws.com

2023-08-03 20:05:53,861 - MainThread - urllib3.connectionpool - DEBUG - https://ec2.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 500 None

2023-08-03 20:05:53,861 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'b6f5bc4f-44ef-4016-84de-2adf68c1be68', 'Cache-Control': 'no-cache, no-store', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'vary': 'accept-encoding', 'Content-Type': 'text/xml;charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 03 Aug 2023 22:35:53 GMT', 'Connection': 'close', 'Server': 'AmazonEC2'}

2023-08-03 20:05:53,862 - MainThread - botocore.parsers - DEBUG - Response body:

b'<?xml version="1.0" encoding="UTF-8"?>\n<Response><Errors><Error><Code>InternalError</Code><Message>An internal error has occurred</Message></Error></Errors><RequestID>b6f5bc4f-44ef-4016-84de-2adf68c1be68</RequestID></Response>'

2023-08-03 20:05:53,862 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.StopInstances: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f95db8f65d0>>

2023-08-03 20:05:53,862 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached.

2023-08-03 20:05:53,862 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.

2023-08-03 20:05:53,862 - MainThread - botocore.hooks - DEBUG - Event after-call.ec2.StopInstances: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f95e29ae010>>

2023-08-03 20:05:53,862 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()

Traceback (most recent call last):

  File "awscli/clidriver.py", line 460, in main

  File "awscli/clidriver.py", line 595, in __call__

  File "awscli/clidriver.py", line 798, in __call__

  File "awscli/clidriver.py", line 929, in invoke

  File "awscli/clidriver.py", line 941, in _make_client_call

  File "awscli/botocore/client.py", line 341, in _api_call

  File "awscli/botocore/client.py", line 697, in _make_api_call

botocore.exceptions.ClientError: An error occurred (InternalError) when calling the StopInstances operation (reached max retries: 2): An internal error has occurred


An error occurred (InternalError) when calling the StopInstances operation (reached max retries: 2): An internal error has occurred
1 Answer
0

Today, I along with few of my colleagues experienced issues while calling API StartInstances, which was intermittent and fixed now. Since you mentioned, you are experiencing this issue since Jul 29th so that doesn't seem related.

I'd suggest you to check AWS Health Dashboard and there you may find, if these instances are affected by any AWS event for any reason. If you have access to log support case, I'd strongly suggest you to reach out AWS support providing these details, someone from support would definitely be able to help you out.

profile pictureAWS
EXPERT
answered 10 months ago
  • Do you have any additional questions, feel free to comment here, happy to assist.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions