I recently performed a yum update on my server running Amazon Linux 2. Ever since then, I cannot connect to the server, and it is failing the status check with the message "Instance reachability check failed." I have read through many posts about this, but I can't seem to find a solution, or the posts asking similar questions are never answered.
I did find some information about ip address "169.254.169.254", which shows up in the log. Someone said something about network issue? But I don't understand why that would be an issue for me, I did not change any of that. I just did the update and rebooted?
Here is a portion of my system log, I am not sure what else to provide.
Starting Postfix Mail Transport Agent...
[ 7.273092] cloud-init[2237]: Cloud-init v. 19.3-46.amzn2.0.1 running 'init' at Mon, 05 Jun 2023 17:19:53 +0000. Up 7.18 seconds.
[ 7.304066] cloud-init[2237]: ci-info: +++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++
[ 7.304323] cloud-init[2237]: ci-info: +--------+------+---------------------------+-----------+-------+-------------------+
[ 7.304448] cloud-init[2237]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 7.304549] cloud-init[2237]: ci-info: +--------+------+---------------------------+-----------+-------+-------------------+
[ 7.304644] cloud-init[2237]: ci-info: | eth0 | True | fe80::12:2ff:fe51:120b/64 | . | link | 02:12:02:51:12:0b |
[ 7.304774] cloud-init[2237]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
[ 7.304901] cloud-init[2237]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 7.305004] cloud-init[2237]: ci-info: +--------+------+---------------------------+-----------+-------+-------------------+
[ 7.305108] cloud-init[2237]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
[ 7.305207] cloud-init[2237]: ci-info: +-------+-----------------+---------+-----------------+-----------+-------+
[ 7.305322] cloud-init[2237]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[ 7.305450] cloud-init[2237]: ci-info: +-------+-----------------+---------+-----------------+-----------+-------+
[ 7.305567] cloud-init[2237]: ci-info: | 0 | 169.254.169.254 | 0.0.0.0 | 255.255.255.255 | eth0 | UH |
[ 7.305673] cloud-init[2237]: ci-info: +-------+-----------------+---------+-----------------+-----------+-------+
[ 7.305767] cloud-init[2237]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[ 7.305864] cloud-init[2237]: ci-info: +-------+-------------+---------+-----------+-------+
[ 7.305964] cloud-init[2237]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[ 7.306059] cloud-init[2237]: ci-info: +-------+-------------+---------+-----------+-------+
[ 7.306187] cloud-init[2237]: ci-info: | 9 | fe80::/64 | :: | eth0 | U |
[ 7.306317] cloud-init[2237]: ci-info: | 11 | local | :: | eth0 | U |
[ 7.306429] cloud-init[2237]: ci-info: | 12 | ff00::/8 | :: | eth0 | U |
[ 7.306553] cloud-init[2237]: ci-info: +-------+-------------+---------+-----------+-------+
[[32m OK [0m] Started Postfix Mail Transport Agent.
[[32m OK [0m] Started MariaDB database server.
[[32m OK [0m] Stopped Run the CVE-2021-44228 hotfix script.
[[32m OK [0m] Started Run the CVE-2021-44228 hotfix script.
[ 57.616696] cloud-init[2237]: Jun 05 17:20:44 cloud-init[2237]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7f4ba27dd0>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]
[ 108.669221] cloud-init[2237]: Jun 05 17:21:35 cloud-init[2237]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [101/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7f4ba24a10>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))]
[ 126.689596] cloud-init[2237]: Jun 05 17:21:53 cloud-init[2237]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7f4ba249d0>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)'))]
[ 127.690978] cloud-init[2237]: Jun 05 17:21:54 cloud-init[2237]: DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled
[[32m OK [0m] Started Initial cloud-init job (metadata service crawler).
[[32m OK [0m] Reached target Cloud-config availability.
[[32m OK [0m] Reached target Network is Online.
Starting Apply the settings specified in cloud-config...
[[32m OK [0m] Started amazon-ssm-agent.
Starting Dynamically Generate Message Of The Day...
Starting System Logging Service...
Starting Notify NFS peers of a restart...
Starting OpenSSH server daemon...
Starting Permit User Sessions...
[[32m OK [0m] Started Notify NFS peers of a restart.
[[32m OK [0m] Started Permit User Sessions.
[[32m OK [0m] Started Command Scheduler.
Starting Terminate Plymouth Boot Screen...
Starting Wait for Plymouth Boot Screen to Quit...
[[32m OK [0m] Started Job spooling tools.
[[32m OK [0m] Created slice User Slice of root.
[[1;31mFAILED[0m] Failed to start Session c1 of user root.
See 'systemctl status session-c1.scope' for details.
[[1;31mFAILED[0m] Failed to start Session c3 of user root.
See 'systemctl status session-c3.scope' for details.
[[1;31mFAILED[0m] Failed to start Session c2 of user root.
See 'systemctl status session-c2.scope' for details.
[[32m OK [0m] Started System Logging Service.
[[32m OK [0m] Started OpenSSH server daemon.
[[32m OK [0m] Removed slice User Slice of root.
[ 128.602842] cloud-init[3263]: Cloud-init v. 19.3-46.amzn2.0.1 running 'modules:config' at Mon, 05 Jun 2023 17:21:54 +0000. Up 128.53 seconds.
[ 129.036365] cloud-init[3322]: Cloud-init v. 19.3-46.amzn2.0.1 running 'modules:final' at Mon, 05 Jun 2023 17:21:55 +0000. Up 128.98 seconds.
[ 129.050401] cloud-init[3322]: Cloud-init v. 19.3-46.amzn2.0.1 finished at Mon, 05 Jun 2023 17:21:55 +0000. Datasource DataSourceNone. Up 129.04 seconds
[ 129.057795] cloud-init[3322]: Jun 05 17:21:55 cloud-init[3322]: cc_final_message.py[WARNING]: Used fallback datasource
Amazon Linux 2
Kernel 4.14.314-238.539.amzn2.x86_64 on an x86_64
ip-172-30-0-37 login: [ 259.040541] hibinit-agent[3321]: Traceback (most recent call last):
[ 259.040995] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 496, in <module>
[ 259.041502] hibinit-agent[3321]: main()
[ 259.042012] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 435, in main
[ 259.050239] hibinit-agent[3321]: if not hibernation_enabled(config.state_dir):
[ 259.050517] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 390, in hibernation_enabled
[ 259.056646] hibinit-agent[3321]: imds_token = get_imds_token()
[ 259.056927] hibinit-agent[3321]: File "/usr/bin/hibinit-agent", line 365, in get_imds_token
[ 259.057515] hibinit-agent[3321]: response = requests.put(token_url, headers=request_header)
[ 259.058573] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 121, in put
[ 259.074207] hibinit-agent[3321]: return request('put', url, data=data, **kwargs)
[ 259.074529] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
[ 259.074993] hibinit-agent[3321]: response = session.request(method=method, url=url, **kwargs)
[ 259.075492] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
[ 259.077613] hibinit-agent[3321]: resp = self.send(prep, **send_kwargs)
[ 259.078188] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
[ 259.079262] hibinit-agent[3321]: r = adapter.send(request, **kwargs)
[ 259.080340] hibinit-agent[3321]: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 419, in send
[ 259.088082] hibinit-agent[3321]: raise ConnectTimeout(e, request=request)
[ 259.088484] hibinit-agent[3321]: requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/api/token (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff5e5b3aa90>: Failed to establish a new connection: [Errno 110] Connection timed out',))
Did you try to ping your problematic instance from one of the 2 others? (to validate that network stack works from local subnet at least up to ICMP layer)