AWS CodeBuild Project Failed - Missing Signature Key caused Docker Image failing to build and push?

The error message missing signature key usually indicates that there is an issue with the Docker image signature verification. This can sometimes occur if there is an issue with the base image being pulled or if there is a misconfiguration in the Docker setup. Let's go through the steps to troubleshoot and resolve this issue.

Troubleshooting Steps Verify Base Image Availability:

Ensure that the base image (ubuntu:latest) is available and accessible from the build environment. Update the Dockerfile:

Simplify the Dockerfile to isolate the problem. Start with a minimal Dockerfile and gradually add commands to identify the problematic step. Updated Dockerfile Start with a simplified Dockerfile and add commands incrementally: Dockerfile:

FROM ubuntu:latest

# Update and install required packages
RUN apt-get update && apt-get install -y python3-pip python3-dev build-essential

# Copy application files
COPY ./service /MythicalMysfitsService

# Set working directory
WORKDIR /MythicalMysfitsService

# Install Python dependencies
RUN pip3 install -r requirements.txt

# Start the Flask service
ENTRYPOINT ["python3"]
CMD ["mythicalMysfitsService.py"]

Update buildspec.yml
Ensure your buildspec.yml is correctly set up to build and push the Docker image:

yaml:
version: 0.2

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/mythicalmysfits/service:latest .
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/mythicalmysfits/service:latest
artifacts:
  files:
    - imagedefinitions.json
  discard-paths: yes

Potential Solutions Check Docker Base Image:

Pull the base image manually to verify if it’s accessible. sh:

docker pull ubuntu:latest

Verify Docker Configuration:

Ensure Docker is configured correctly in your AWS CodeBuild environment. Check if there are any restrictions or proxy settings that might affect the image pull. Update CodeBuild Environment:

Ensure that the AWS CodeBuild environment is using a Docker-enabled image, such as the aws/codebuild/standard:5.0 or a similar image. Example Docker Build Command Ensure that the Docker build command in your local environment works as expected. This helps verify that there are no issues with the Dockerfile itself. sh:

docker build -t my-account-id.dkr.ecr.us-west-2.amazonaws.com/mythicalmysfits/service:latest .

Check IAM Permissions Ensure that the IAM role used by CodeBuild has the necessary permissions to pull images from Docker Hub and push images to Amazon ECR.

Final Notes If the issue persists, you can:

Check Docker Daemon Logs: Look into the Docker daemon logs for more detailed error messages. AWS Support: If all else fails, consider reaching out to AWS Support for assistance with the specific error message you are encountering. By following these steps, you should be able to troubleshoot and resolve the missing signature key issue in your Docker build process within AWS CodeBuild.

we faced the same issue recently and no solution.

[Container] 2024/08/08 08:53:43.882110 Running command docker build --build-arg AWS_REGION=$AWS_DEFAULT_REGION -t $ACCOUNT_DKR_URI/$IMAGE_NAME:latest -f codebuild/cb-build-deploy/Dockerfile .
Sending build context to Docker daemon  449.5MB
Step 1/28 : FROM amazonlinux:1
missing signature key
[Container] 2024/08/08 08:53:46.751075 Command did not exit successfully docker build --build-arg AWS_REGION=$AWS_DEFAULT_REGION -t $ACCOUNT_DKR_URI/$IMAGE_NAME:latest -f codebuild/cb-build-deploy/Dockerfile . exit status 1
[Container] 2024/08/08 08:53:46.754460 Phase complete: BUILD State: FAILED
[Container] 2024/08/08 08:53:46.754473 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: docker build --build-arg AWS_REGION=$AWS_DEFAULT_REGION -t $ACCOUNT_DKR_URI/$IMAGE_NAME:latest -f codebuild/cb-build-deploy/Dockerfile .. Reason: exit status 1
[Container] 2024/08/08 08:53:46.787816 Entering phase POST_BUILD
[Container] 2024/08/08 08:53:46.788663 Running command echo Build completed on `date`
Build completed on Thu Aug 8 08:53:46 UTC 2024

An update to my dilemma: So, I did a series of unanticipated steps that involved signing the docker image that was stored in my Amazon ECR private repository, downloading and installing pass, docker-credential-pass, docker-credential-helpers packages, and a couple of other steps. Of course, before doing all of this, I had to fulfill a few prerequisites which included downloading, installing, and verifying the container-signing tools of 1) AWS Signer plugin for Notation and 2) Notation, an open source supply chain security program. After that, I created an AWS Signer signing profile that uses the Notation-OCI-SHA384-ECDSA signing platform, followed by using AWS CLI to authenticate the Notation CLI to my Amazon ECR private registry. Once those procedures were done, I went through with the signing protocol by using Notation CLI to sign the container image, according to both the repository name and the SHA digest, all of which resulted in a positive message saying "Successfully signed aws-account-id.dkr.ecr.region.amazonaws.com/repository-name/service@sha256....." (removed confidential details).

ec2-user:~/environment $ git clone https://github.com/docker/docker-credential-helpers.git
Cloning into 'docker-credential-helpers'...
remote: Enumerating objects: 1732, done.
remote: Counting objects: 100% (584/584), done.
remote: Compressing objects: 100% (194/194), done.
remote: Total 1732 (delta 492), reused 390 (delta 390), pack-reused 1148
Receiving objects: 100% (1732/1732), 721.08 KiB | 5.42 MiB/s, done.
Resolving deltas: 100% (971/971), done.

ec2-user:~/environment $ git clone https://git.zx2c4.com/password-store
Cloning into 'password-store'...
remote: Enumerating objects: 2616, done.
remote: Total 2616 (delta 0), reused 0 (delta 0), pack-reused 2616 (from 1)
Receiving objects: 100% (2616/2616), 411.06 KiB | 674.00 KiB/s, done.
Resolving deltas: 100% (1512/1512), done.

ec2-user:~/environment $ cd password-store
ec2-user:~/environment/password-store (master) $ ls -a
.  ..  .git  .gitignore  COPYING  INSTALL  Makefile  README  contrib  man  src  tests
ec2-user:~/environment/password-store (master) $ sudo make install
'man/pass.1' -> '/usr/share/man/man1/pass.1'
'src/completion/pass.bash-completion' -> '/usr/share/bash-completion/completions/pass'
'src/completion/pass.zsh-completion' -> '/usr/share/zsh/site-functions/_pass'
install: creating directory '/usr/lib/password-store'
install: creating directory '/usr/lib/password-store/extensions'
'src/.pass' -> '/usr/bin/pass'

ec2-user:~/environment $ wget https://github.com/docker/docker-credential-helpers/releases/download/v0.6.0/docker-credential-pass-v0.6.0-amd64.tar.gz && tar -xf docker-credential-pass-v0.6.0-amd64.tar.gz && chmod +x docker-credential-pass && sudo mv docker-credential-pass /usr/local/bin/
--2024-07-14 21:53:15--  https://github.com/docker/docker-credential-helpers/releases/download/v0.6.0/docker-credential-pass-v0.6.0-amd64.tar.gz
Resolving github.com (github.com)... 140.82.116.3
Connecting to github.com (github.com)|140.82.116.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/51309425/c45e280a-8d93-11e7-956b-7c320236531b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240714%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240714T215315Z&X-Amz-Expires=300&X-Amz-Signature=9b20194d5cab0ccf39f23a40d354db8f02b9a75bd29719aa1728a8acf4dc1d4d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=51309425&response-content-disposition=attachment%3B%20filename%3Ddocker-credential-pass-v0.6.0-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2024-07-14 21:53:15--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/51309425/c45e280a-8d93-11e7-956b-7c320236531b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240714%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240714T215315Z&X-Amz-Expires=300&X-Amz-Signature=9b20194d5cab0ccf39f23a40d354db8f02b9a75bd29719aa1728a8acf4dc1d4d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=51309425&response-content-disposition=attachment%3B%20filename%3Ddocker-credential-pass-v0.6.0-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2539520 (2.4M) [application/octet-stream]
Saving to: ‘docker-credential-pass-v0.6.0-amd64.tar.gz’

docker-credential-pass-v0.6.0-amd64.tar.gz                100%[====================================================================================================================================>]   2.42M  --.-KB/s    in 0.07s

2024-07-14 21:53:16 (32.5 MB/s) - ‘docker-credential-pass-v0.6.0-amd64.tar.gz’ saved [2539520/2539520]

ec2-user:~/environment $ gpg2 --gen-key

GnuPG needs to construct a user ID to identify your key.

Real name: myname
Email address: your_email@address.com
You selected this USER-ID:
    "myname <your_email@address.com>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: revocation certificate stored as '/home/ec2-user/.gnupg/openpgp-revocs.d/key......rev'
public and secret key created and signed.

ec2-user:~/environment $ pass init myname
Password store initialized for myname
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: next trustdb check due at 2026-07-14
docker-credential-helpers/docker-pass-initialized-check: reencrypting to 111222333444555666

ec2-user:~/environment $ pass init A1B2C3D4E5F6G7H8I9J10
Password store initialized for A1B2C3D4E5F6G7H8I9J10
ec2-user:~/environment $ pass insert docker-credential-helpers/docker-pass-initialized-check
ec2-user:~/environment $ pass show docker-credential-helpers/docker-pass-initialized-check
pass is initialized
ec2-user:~/environment $ docker-credential-pass list
{}

ec2-user:~/environment $ aws ecr get-login-password --region REGION | docker login --username AWS --password-stdin AWS-ACCOUNT-ID.dkr.ecr.REGION.amazonaws.com
Login Succeeded
ec2-user:~/environment $ aws ecr get-login-password --region REGION | notation login --username AWS --password-stdin AWS-ACCOUNT-ID.dkr.ecr.REGION.amazonaws.com
Login Succeeded

ec2-user:~/environment/aws-modern-application-workshop/module-2/app (python) $ notation sign AWS-ACCOUNT-ID.dkr.ecr.REGION.amazonaws.com/repository-name@sha256:ca78e5f730f9a789ef8c63bb55275ac12dfb9e8099e6EXAMPLE --plugin "com.amazonaws.signer.notation.plugin" --id "arn:aws:signer:REGION:AWS-ACCOUNT-ID:/signing-profiles/ecrSigningProfileName"
Successfully signed AWS-ACCOUNT-ID.dkr.ecr.REGION.amazonaws.com/repository-name/service@sha256:a16ce7011918939a3383a94e7bc68a6f4113c05286a00af40a7518f7981c3260

Yet, even after doing all of this, and using the git add. git commit -m "I changed the age of one of the mysfits." git push commands to apply these changes, I still come up with the same issue that I encountered at the beginning, even after seeing the signed container image on the AWS console, so I'm honestly confused on what is going on here and why I'm still getting the same error?

Quick Update:

So, unbeknownst to me all this time, I did not realize that the Docker version itself might be a factor into this. Upon having the CI/CD pipeline run docker info and docker version, based on the buildspec.yml, I ended up with these responses:

[Container] 2024/07/15 02:17:41.048777 Running command docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.291-218.527.amzn2.x86_64
Operating System: Ubuntu 14.04.5 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.644GiB
Name: 6c5b18c84c6b
ID: PKFE:D5BS:JM7G:U6EI:CH5R:3F5S:4MPU:WRKH:HEOU:6RTS:XNJE:KETV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[Container] 2024/07/15 02:17:41.067564 Running command docker --version
Docker version 17.09.0-ce, build afdb6d4

The Docker version that it brought out is outdated, since the Docker version used in my Amazon EC2 environment has the Docker version 25.0.3, build 4debf41 set:

$ docker info

Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.0.0+unknown
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 3
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 64b8a811b07ba6288238eefc14d898ee0b5b99ba
 runc version: 4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.94-99.176.amzn2023.x86_64
 Operating System: Amazon Linux 2023.5.20240624
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 949.5MiB
 Name: ip-172-31-0-208.us-west-2.compute.internal
 ID: ae6963b0-cafe-4e13-9c70-90bd570ced23
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Would this likely relate to why I'm getting the error I'm encountering?