ECR login token expiry - reauthentication suggestions

0

We are seeing random ECR docker login token expiry errors in our Jenkins pipelines.

docker: Error response from daemon: pull access denied for my-account.dkr.ecr.us-east-1.amazonaws.com/my-project/myapp/myimage:myversion, repository does not exist or may require 'docker login': denied: Your authorization token has expired. Reauthenticate and try again.

As per AWS document, https://docs.aws.amazon.com/AmazonECR/latest/public/public-troubleshooting.html, we need to reauthenticate the ecr login. We use ecr get-login within our Jenkinsfile to run activities within a docker container as below,

sh '$(aws ecr get-login --no-include-email --region us-east-1)'
            withDockerContainer(image: 'myaccount.dkr.ecr.us-east-1.amazonaws.com/my-project/myapp/my-image:my-version'){

We were planning to update the ecr login section over a try-catch and ensure the reauthentication is done in the catch block. We would like to get a confirmation if there are any other better ways of handling this error.  

try {
                        sh '$(aws ecr get-login --no-include-email --region us-east-1)'
                    } catch (Exception e) {
                        sh 'docker logout "myaccount.dkr.ecr.us-east-1.amazonaws.com"'
                        sh '$(aws ecr get-login --no-include-email --region us-east-1)'
                    }
asked 10 months ago948 views
1 Answer
1
Accepted Answer

If you set up a Jenkinsfile like this, don't you log in every time?
https://docs.aws.amazon.com/AmazonECR/latest/userguide/Registries.html
https://www.jenkins.io/doc/book/pipeline/docker/

pipeline{
    agent any
    stages{
        stage('Hoge') {
            steps{
                script{
                    docker.withRegistry("https://${AWSAccountID}.dkr.ecr.${ECR Region}.amazonaws.com") {
                        sh("`aws ecr get-login --no-include-email --region ${ECR Region}`")
                        docker.image('my-custom-image').inside {
                          sh("echo 'hoge'")
                        }
                    }
                }
            }
        }
    }
}
profile picture
EXPERT
answered 10 months ago
profile pictureAWS
EXPERT
reviewed 10 months ago
  • AFAIK, this one is better than the try / catch: I don't see why the second login in cache block would succeed if initial on just failed. They will be quasi-simultanous. The issue that you described initially is more of a timeout happening long after initial login so that will be ineffective.

  • The token expiry happens quite randomly. It does not happen always. So if indeed the token has expired, we need to be doing reauthentication as per AWS suggestion. Hence, believed that the try catch will ensure that it will perform the reauthentication in case of failure.

  • Are you suggesting that changing to docker.withRegistry will solve the issue for us? May I know why the withDockerContainer gives back such issues?

  • I'm not sure why withDockerContainer would cause the token expiration issue, but I believe docker.withRegistry logs you in every time you run with a build.

  • Understood. I will give this a try. There is also a recommendation to cleanup current user docker credentials using sh 'rm ~/.docker/config.json || true' on https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/withdockerregistry-step-fails-with-amazon-ecr and https://plugins.jenkins.io/amazon-ecr/. If I include that it is introducing delays in the docker image download and times out. Is the docker credentials cleanup necessary?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions