How are you keeping your CFN template up-to-date with ECS image tags?

Question

We use CFN to define all of our infrastructure and when I created the ECS TaskDefinition I used this container image "1111122223333444.dkr.ecr.us-east-1.amazonaws.com/my_app".  This resource uses the "latest" container image in our ECR repo.  However, I don't want to use the "latest" container image, I want to use a specific image (ie. "1111122223333444.dkr.ecr.us-east-1.amazonaws.com/my_app:AAA123").

My deployment scripts automatically create new TaskDefinitions whenever there is new code, so that works well.  However, every time I change my ECS CFN template (ie. add a new environment variable) it creates a new TaskDefinition and resets the image to "1111122223333444.dkr.ecr.us-east-1.amazonaws.com/my_app".

Now we're a bit out of sync; the "LATEST" ECS TaskDefinition is pointing to the latest container image, but the latest code is in "1111122223333444.dkr.ecr.us-east-1.amazonaws.com/my_app:BBB456".

One possible solution would be to store the current container image tag in an SSM parameter and reference that in the CFN template.  That way, whenever a new container image is created CFN knows about it and can reference the new image.  However, this seems pretty round-a-bout and prone to breakage.  I also don't want to automatically update the CFN stack whenever we push out new code.  CFN can be a bit temperamental and I don't want to have it do something weird or get into an unknown state.

JohnPreston · Accepted Answer

Hello.
For having managed 1000s of deployments to ECS with CFN for the past 4y, I recommend **never** to use the image tags, and instead always use the image digest (I usually create a docker label that uses the image tag at the time, for ease of identification).

The simple reason for that is that you want your image deployed to be immutable, if you use the tags, these can change, and if you use `latest` as the default tag, then you will have a very hard time doing any form of version control of what is the version of the software actually deployed. Particularly, if you follow a pipeline like dev/stg/prod, then if you update on build the `latest` image, new prod containers might be pulling the image you are testing in dev, less than optimal.

I am the maintainer of [ECS Compose-X](https://docs.compose-x.io) which generates the CFN templates from docker compose files during the pipeline, and I run it after the new image was built and uploaded to ECR (that way I get the sha etc., and also optionally perform an ECR vuln scan against the image).

By default, I tag the new images built in the process with the first 7 git commit characters, making it easy to link back to the code if needed, but we always, always, use the SHA for the main and sidecar containers, to ensure that we are deploying in prod the same thing that was used and tested in any prior environments.

I hope that helps :)

EDIT1: The image to use for the service/container is always a parameter to the CFN stack, as to allow for break glass solutions when you need to revert or something like that, without having to wait for a pipeline to run, which can sometimes take quite a while.

re:Post Agent · Answer

To address your concern about keeping your CloudFormation (CFN) template up-to-date with specific ECS image tags, there are a few strategies you can implement:

1. Use Parameters in your CloudFormation template:
   Instead of hardcoding the image URI in your template, you can use a parameter. This allows you to specify the exact image tag when you update your stack, without changing the template itself.

2. Implement image tag immutability:
   In Amazon ECR, you can set your repository to use immutable image tags. This prevents the "latest" tag from being overwritten and ensures that each image push creates a new tag. This practice helps maintain version consistency and prevents unintended updates.

3. Use container image digests:
   Instead of using image tags, you can use image digests in your task definition. Image digests are unique identifiers for a specific image version and don't change, unlike tags which can be overwritten.

4. Force new deployments:
   When updating your service, you can use the "forceNewDeployment" option. This ensures that even if you're using the same tag (like "latest"), ECS will pull the current image from your repository when starting new tasks.

5. Implement a CI/CD pipeline:
   Set up a pipeline that automatically updates your CloudFormation stack with the latest image tag whenever a new image is pushed to your ECR repository. This keeps your infrastructure code in sync with your application code.

6. Use separate templates:
   Consider separating your ECS task definition into a separate template from your other infrastructure. This allows you to update the task definition (including the image tag) independently of other infrastructure changes.

By implementing these strategies, you can ensure that your ECS tasks are always using the intended container image, even when you make changes to your CloudFormation template for other reasons (like adding environment variables). This approach provides better control over your deployments and helps maintain consistency between your infrastructure code and your application code.

Remember, when you update your service, ECS resolves the container image tag to a specific digest. This means that even if you update the image associated with a tag in your repository, your running tasks won't be affected until you perform a new deployment. This behavior helps ensure version consistency across your service.
      
**Sources**  
[Use new container image in Amazon ECS | AWS re:Post](https://repost.aws/knowledge-center/older-container-image-ecs)  
[ContainerImage - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerImage.html)  
[UpdateService - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_UpdateService.html)

How are you keeping your CFN template up-to-date with ECS image tags?

Relevant content