AMI Customizations disappear mysteriously after boot

0

Over the past few weeks, I've been seeing a very strange issue. At first I thought I must have been missing a step in my AMI creation process, but last night it happened again. Here's what we did and experienced:

Used a RHEL-7 AMI as a base, and we installed our own codebase and libraries on there. From that we created an AMI. As we've discovered more dependencies and updated the codebase, we've updated the AMI to include the new stuff.

Last night then, I booted up an instance from the AMI. We ssh'd in (via AWS SSM Session Manager) and checked and everything was working. Randomly about 40 minutes later it stopped pinging into our pingdom dashboard.

In trying to investigate: we couldn't start an AWS SSM session. Instead we had to ssh in via the old jump host.
On the instance intself -- the ssm-agent is not even present, not running. The custom config files (crontab and scoutdaemon) are no longer even on the system! The instance just looks like some plain RHEL from a couple of months ago, not the one we booted last night.

Has anyone seen this happen before? What's the underlying cause / what am I doing wrong?

This reduces our confidence in the stability of this

seemant
demandé il y a 4 ans256 vues
2 réponses
0

Hello seemant,

Could you please share the specific instance IDs, the region in question, the times and dates related to each change (at least approximate if you do not know the exact times, and also please include the time zone), the AMI IDs with the steps that you have taken all chronologically described? We do not have visibility into the instances internally, nor we can see into the AMIs when they are not public, but we might be able to help you to pinpoint the issue if we have more specific information. Further, if you have a Premium Support plan, you can create a new support case to have this all reviewed within the SLA.

Thank you for your understanding.

Regards,
awstomas

AWS
répondu il y a 4 ans
0

Thank you. I'll close this post for now as I'm no longer generating custom AMIs at the moment

seemant
répondu il y a 4 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions