By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Took image from a working instance (multiple times), created new instance from that AMI, it won't start up.

0

We had an OS upgrade gone wrong on an EC2 instance. The day before the attempted upgrade we took an image. We have multiple older images as well. The upgrade caused some issue, but we weren't concerned. We stopped the instance and then went into AMI and launched a new instance from the image that was taken a day earlier. All of the properties matched the original. The problem...it doesn't start up. I would attach th log, but I don't see an attach option. I've pasted what looks like the important parts. Here's the big gotcha in all of this, this instance never had a local user with a password created (if it did, no one working on the project is aware of one). The only way we ever got in was via key. So as far as I know, the EC2 Serial Console doesn't help, correct?

So, any ideas what is happening and what we can do (if anything) to get to the machine? I'm not sure what the point of the multiple backups were if the restoring from backup doesn't allow us to get in. Any help is appreciated. Thank you very much.

[    8.330811] cloud-init[526]: Traceback (most recent call last):
         Starting Network Time Synchronization...
[   10.416376] cloud-init[526]:   File "/usr/bin/cloud-init", line 9, in <module>
[   11.067454] cloud-init[526]:     load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')()
[   11.708544] cloud-init[526]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 488, in load_entry_point
[   11.708754] cloud-init[526]:     return get_distribution(dist).load_entry_point(group, name)
[   11.709207] cloud-init[526]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2861, in load_entry_point
[   11.709638] cloud-init[526]:     return ep.load()
[   11.710096] cloud-init[526]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2461, in load
[   11.710547] cloud-init[526]:     return self.resolve()
[   11.711009] cloud-init[526]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2467, in resolve
[   11.711454] cloud-init[526]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
[   11.711906] cloud-init[526]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 29, in <module>
[   11.712820] cloud-init[526]:     from cloudinit import stages
[   11.713250] cloud-init[526]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 21, in <module>
[   11.713696] cloud-init[526]:     from cloudinit.handlers.cloud_config import CloudConfigPartHandler
[   11.714150] cloud-init[526]:   File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 11, in <module>
[   11.714595] cloud-init[526]:     import jsonpatch
[   11.715057] cloud-init[526]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 114, in <module>
[   11.715494] cloud-init[526]:     json.load = get_loadjson()
[   11.715940] cloud-init[526]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 108, in get_loadjson
[   11.716393] cloud-init[526]:     argspec = inspect.getargspec(json.load)
[   11.716836] cloud-init[526]:   File "/usr/lib/python3.8/inspect.py", line 1083, in getargspec
[   11.717285] cloud-init[526]:     raise ValueError("Function has keyword-only parameters or annotations"
[   11.717755] cloud-init[526]: ValueError: Function has keyword-only parameters or annotations, use inspect.signature() API which can support them
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Network Time Synchronization.
[FAILED] Failed to start Initial cloud-init job (pre-networking).

[FAILED] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
[  OK  ] Reached target Network.
         Starting Initial cloud-ini… (metadata service crawler)...
[   13.856112] cloud-init[832]: Traceback (most recent call last):
[   13.858879] cloud-init[832]:   File "/usr/bin/cloud-init", line 9, in <module>
[   13.861919] cloud-init[832]:     load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')()
[   13.865587] cloud-init[832]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 488, in load_entry_point
[   13.869746] cloud-init[832]:     return get_distribution(dist).load_entry_point(group, name)
[   13.873157] cloud-init[832]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2861, in load_entry_point
[   13.877357] cloud-init[832]:     return ep.load()
[   13.879650] cloud-init[832]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2461, in load
[   13.883497] cloud-init[832]:     return self.resolve()
[   13.885914] cloud-init[832]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2467, in resolve
[   13.889862] cloud-init[832]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
[   13.893406] cloud-init[832]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 29, in <module>
[   13.897181] cloud-init[832]:     from cloudinit import stages
[   13.899730] cloud-init[832]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 21, in <module>
[   13.903479] cloud-init[832]:     from cloudinit.handlers.cloud_config import CloudConfigPartHandler
[   13.906938] cloud-init[832]:   File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 11, in <module>
[   13.911055] cloud-init[832]:     import jsonpatch
[   13.913300] cloud-init[832]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 114, in <module>
[   13.916858] cloud-init[832]:     json.load = get_loadjson()
[   13.919317] cloud-init[832]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 108, in get_loadjson
[   13.923062] cloud-init[832]:     argspec = inspect.getargspec(json.load)
[   13.925891] cloud-init[832]:   File "/usr/lib/python3.8/inspect.py", line 1083, in getargspec
[   13.929291] cloud-init[832]:     raise ValueError("Function has keyword-only parameters or annotations"
[   13.932933] cloud-init[832]: ValueError: Function has keyword-only parameters or annotations, use inspect.signature() API which can support them
[FAILED] Failed to start Initial cl…ob (metadata service crawler).
See 'systemctl status cloud-init.service' for details.

[  OK  ] Started Service for snap a…on-ssm-agent.amazon-ssm-agent.
[   16.165094] cloud-init[1162]: Traceback (most recent call last):
[   16.169623] cloud-init[1162]:   File "/usr/bin/cloud-init", line 9, in <module>
[   16.174169] cloud-init[1162]:     load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')()
[   16.180950] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 488, in load_entry_point
[   16.188357] cloud-init[1162]:     return get_distribution(dist).load_entry_point(group, name)
[   16.194035] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2861, in load_entry_point
[   16.201210] cloud-init[1162]:     return ep.load()
[   16.205683] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2461, in load
[   16.212624] cloud-init[1162]:     return self.resolve()
[   16.217400] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2467, in resolve
[   16.223954] cloud-init[1162]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
[   16.230103] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 29, in <module>[FAILED] Failed to start Apply the …ngs specified in cloud-config.
See 'systemctl status cloud-config.service' for details.

[   16.243759] cloud-init[1162]:     from cloudinit import stages
[   16.247516] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 21, in <module>
[   16.252672] cloud-init[1162]:     from cloudinit.handlers.cloud_config import CloudConfigPartHandler
[   16.257065] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 11, in <module>
[   16.261911] cloud-init[1162]:     import jsonpatch
[   16.264874] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 114, in <module>
[   16.269285] cloud-init[1162]:     json.load = get_loadjson()
[   16.272249] cloud-init[1162]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 108, in get_loadjson
[   16.276645] cloud-init[1162]:     argspec = inspect.getargspec(json.load)
[   16.280090] cloud-init[1162]:   File "/usr/lib/python3.8/inspect.py", line 1083, in getargspec
[   16.284180] cloud-init[1162]:     raise ValueError("Function has keyword-only parameters or annotations"
[   16.288301] cloud-init[1162]: ValueError: Function has keyword-only parameters or annotations, use inspect.signature() API which can support them
[  OK  ] Stopped Service for snap a…on-ssm-agent.amazon-ssm-agent.
[  OK  ] Started Service for snap a…on-ssm-agent.amazon-ssm-agent.
[  OK  ] Stopped Service for snap a…on-ssm-agent.amazon-ssm-agent.
[  OK  ] Started Service for snap a…on-ssm-agent.amazon-ssm-agent.
[  OK  ] Stopped Service for snap a…on-ssm-agent.amazon-ssm-agent.
[  OK  ] Started Service for snap a…on-ssm-agent.amazon-ssm-agent.
[  OK  ] Stopped Service for snap a…on-ssm-agent.amazon-ssm-agent.
[FAILED] Failed to start Service fo…on-ssm-agent.amazon-ssm-agent.
See 'systemctl status snap.amazon-ssm-a….amazon-ssm-agent.service' for details.


Ubuntu 16.04.6 LTS ip-10-0-11-92 ttyS0

ip-10-0-11-92 login: [   54.009551] cloud-init[1620]: Traceback (most recent call last):
[   54.011210] cloud-init[1620]:   File "/usr/bin/cloud-init", line 9, in <module>
[   54.011865] cloud-init[1620]:     load_entry_point('cloud-init==19.4', 'console_scripts', 'cloud-init')()
[   54.013561] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 488, in load_entry_point
[   54.013725] cloud-init[1620]:     return get_distribution(dist).load_entry_point(group, name)
[   54.013915] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2861, in load_entry_point
[   54.014097] cloud-init[1620]:     return ep.load()
[   54.014285] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2461, in load
[   54.014475] cloud-init[1620]:     return self.resolve()
[   54.014670] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2467, in resolve
[   54.014860] cloud-init[1620]:     module = __import__(self.module_name, fromlist=['__name__'], level=0)
[   54.015048] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 29, in <module>
[   54.015242] cloud-init[1620]:     from cloudinit import stages
[   54.015426] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 21, in <module>
[   54.015617] cloud-init[1620]:     from cloudinit.handlers.cloud_config import CloudConfigPartHandler
[   54.015811] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 11, in <module>
[   54.016050] cloud-init[1620]:     import jsonpatch
[   54.016211] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 114, in <module>
[   54.016398] cloud-init[1620]:     json.load = get_loadjson()
[   54.016590] cloud-init[1620]:   File "/usr/lib/python3/dist-packages/jsonpatch.py", line 108, in get_loadjson
[   54.016778] cloud-init[1620]:     argspec = inspect.getargspec(json.load)
[   54.016968] cloud-init[1620]:   File "/usr/lib/python3.8/inspect.py", line 1083, in getargspec
[   54.017170] cloud-init[1620]:     raise ValueError("Function has keyword-only parameters or annotations"
[   54.017414] cloud-init[1620]: ValueError: Function has keyword-only parameters or annotations, use inspect.signature() API which can support them
  • dmoczulski
    a year ago

    I have an update. We restored from a 2021 backup and that one works. Being 2 years old doesn't help much, but it's something. But clearly the instance has been running and being used daily for those 2 years.

Topics
Compute
Tags
Amazon EC2
Language
English
dmoczulski
asked a year ago214 views
1 Answer
0

You can use the snapshot and create a volume. Then attach that volume to a running instance that you can log in to. Then you can troubleshoot the data on the volume further. Or transfer any updated data to the running instance.

profile pictureAWS
EXPERT
kentrad
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content