How do I resolve image build pipeline execution error "Unable to bootstrap TOE" in Image Builder?

5 minute read
0

My image build pipeline execution fails with the error "Unable to bootstrap TOE" in EC2 Image Builder. How do I resolve this?

Short description

In the build stage of an image build pipeline, the Amazon Elastic Compute Cloud (Amazon EC2) build instance installs and runs the AWS Task Orchestrator and Executor (AWS TOE) bootstrap script bootstrap.sh. If the Amazon EC2 build instance can't install or run the bootstrap script, then you receive an error in the EC2 Image Builder console.

Example error message:

SSM execution '7729e837-22d4-49dd-bb60-cc2c835a2625' failed for image arn: 'arn:aws:imagebuilder:us-west-2:account-ID:image/test/1.0.0/1' with 
status= 'Failed' in state = 'BUILDING' and failure message = 'Unable to bootstrap TOE'

Image Builder uses AWS Systems Manager Automation to build custom images. Automation usually fails at the RunBuildScriptsWithLogging or RunBuildScriptsWithoutLogging step.

The following are common reasons for bootstrap script failures:

  • The EC2 build instance doesn't have access to Amazon Simple Storage Service (Amazon S3).
  • The base Amazon Machine Image (AMI) is CIS hardened, and the default /tmp directory is mounted with noexec.

Resolution

To troubleshoot the issue, use the SSM Automation execution ID from the error message.

  1. Open the EC2 Image Builder console.
  2. Choose the failed build pipeline.
  3. Note the SSM Automation execution ID from the error message. The ID from the preceding example error message is 7729e837-22d4-49dd-bb60-cc2c835a2625.
  4. Open the AWS Systems Manager console.
  5. In the navigation pane, choose Automations.
  6. Select the automation that's associated with the SSM Automation execution ID.
  7. Choose the ID for the RunBuildScriptsWithoutLogging step to get more details about the bootstrap script failure.

The build instance doesn't have access to Amazon S3

During the RunBuildScriptsWithoutLogging step, you receive the following error message with No such file or directory:

{"Status":"Failed","ResponseCode":1,"Output":"Waiting for Cloud-init to initialize ...
Cloud-init fails to initialize ... waiting another 5 minutes for the instance to stabilize
/var/lib/amazon/ssm/i-0ad513xxxxxxx/document/orchestration/bb0d8b53-994f-4c47-8297-e852a484501d/awsrunShellScript/0.awsrunShellScript/_script.sh: 
line 49: /tmp/imagebuilder/TaskOrchestratorAndExecutor/bootstrap.sh: No such file or directory
{\"failureMessage\":\"Unable to bootstrap TOE\"}

The preceding error means that the bootstrap.sh file doesn't exist on the build instance. This error occurs when the build instance is in a private subnet and can't download the AWS TOE bootstrap script.

To resolve this issue, check whether the Amazon S3 gateway endpoint com.amazonaws.region.s3 is in the same Amazon Virtual Private Cloud (Amazon VPC) as the pipeline's infrastructure configuration. If the endpoint is in the same Amazon VPC, then verify that it's associated with the correct route table. The endpoint must be associated with the route table that's used by the subnet that's specified in the infrastructure configuration.

Also, verify that the security group that's attached to the build instance allows outbound HTTPS (port 443) connectivity to the S3 public IP address. The build instance must allow outbound HTTPS connectivity to download the bootstrap script from the Image Builder S3 bucket. Verify the S3 public IP address range that corresponds to the AWS Region of the pipeline.

Note: Amazon S3 interface endpoints don't support the private DNS feature. For more information, see Access an AWS service using an interface VPC endpoint.

Also, make sure that the endpoint’s policy allows the GetObjects action from the Image Builder S3 bucket ec2imagebuilder-toe-REGION-prod. The policy has full access by default.

The /tmp directory is mounted with noexec (applicable for only Linux AMI build)

During the RunBuildScriptsWithoutLogging step, you receive the following error message with Permission denied:

{"Status":"Failed","ResponseCode":1,"Output":"Waiting for Cloud-init to initialize ...
/var/lib/amazon/ssm/i-0ad513xxxxxxx/document/orchestration/634699d5-3b04-4152-aab3-33d6981524cd/awsrunShellScript/0.awsrunShellScript/_script.sh:
line 49: /tmp/imagebuilder/TaskOrchestratorAndExecutor/bootstrap.sh: Permission denied
{\"failureMessage\":\"Unable to bootstrap TOE\"}

The preceding error means that the build instance can't run the bootstrap.sh script because of operating system permissions issues. This error occurs when the default /tmp directory is mounted with the noexec option. The noexec mount option prevents binaries from running on the mounted file system.

To check if the /tmp directory is mounted with noexec, launch a test instance from the base AMI that's specified in the image recipe.

Run the following command:

sudo mount | grep -E '/tmp.*noexec'

Output:

dev/xvdf on /tmp type ext4 (rw,noexec,relatime,data=ordered)

The preceding output confirms that the /tmp directory is mounted as noexec. To resolve this issue, either change the working directory in the image recipe, or remove the noexec option from the directory in the base AMI.

To modify the working directory parameter in the image recipe, create a new image recipe version. Then, edit the pipeline to use the new version. Make sure that the new directory is already in the AMI and not configured to mount with the noexec option.

To remove the noexec option from the /tmp directory:

  1. Manually launch an instance from the base AMI that's specified in the image recipe.
  2. In the /etc/fstab file, remove the noexec option from the entry that's associated with the /tmp or working directory.
  3. Create a new AMI from the instance to use as the base AMI of the image recipe.

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago