AWS logs in docker-compose with temporary credentials

1

Hi,

I am trying to convert my existing setup with hard coded credentials on an edge device to temporary credentials using the systems manager agent. In my existing set up I use aws-log as a logging driver for my docker-compose services, this will automatically create and maintain log streams to a defined log group. Howver, when using the temporary credentials granted from installing the systems manager agent on the edge device, I get an error. I pipe a log in password into my docker deamon as such:
sudo aws ecr get-login-password --profile ssm-iot-gg --region eu-west-1 | sudo docker login --username AWS --password-stdin [account id].dkr.ecr.eu-west-1.amazonaws.com

When trying to start my docker-compose environment I get an error for each service trying to create a log stream:
... failed to create task for container: failed to initialize logging driver: failed to create Cloudwatch log stream: operation error CloudWatch Logs: CreateLogStream, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded

To me it seems that this logging driver might not be compatible with temporary credentials, as the error reads `failed to refresh cached credentials.

Hope someone has an idea of i solution,

Thanks in advance,

Lasse

3 Answers
1

similar issue, but with 404 IAM roles are set and should be working fine, credentials are added in the command line, but nevertheless I get this

Error response from daemon: failed to create task for container: failed to initialize logging driver: failed to create Cloudwatch log stream: operation error CloudWatch Logs: CreateLogStream, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed

answered a year ago
0

Hi, I believe that you should read following blog post to solve you issue in trying to supply AWS credentials to Docker runtime: https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/

Following section is key one for your use case: read it and following to create proper context with AWS credentials.

A note on the AWS credentials: if you are already familiar with AWS you probably already 
have your AWS CLI environment ready with either a default or named profiles. That’s fine, 
the Docker CLI can use those credentials. If not, the Docker workflow will allow you to either 
read the environment variables with your AWS credentials (AWS_ACCESS_KEY_ID and 
AWS_SECRET_ACCESS_KEY) or it will ask for those credentials and will store the credentials 
for you (in $HOME/.aws/credentials).

Best, Didier

profile pictureAWS
EXPERT
answered a year ago
  • I have the same issue, even while populating AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN in the environment, after running and sts assume-role. When running log operation in my terminal session all goes well, but on starting a container the credentials are not picked up even when passing all the environment to docker with --env-file. I suspect the docker daemon doesn't care about the environment variables once it's started, something like that.

0

I solved it by creating a new IAM user with access only to create logstreams and configure this user as my default on the host. Then running sudo aws ecr get-login-password --region eu-west-1 | sudo docker login --username AWS --password-stdin [account id].dkr.ecr.eu-west-1.amazonaws.com will pipe the valid credentials to the docker daemon.

answered a year ago
  • I guess this requires an ECR access, but I'm hosting my images on dockerhub. nevertheless, it should work without any docker repository :/

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions