Skip to content

Does EMR Serverless refresh images from ECR if they change?

0

I use EMR Serverless applications and use custom image based on EMR 7.6.0. That works fine with my pipelines. I am currently using a single tag for my images that I store in ECR. I am quite confident that EMR never checks for updates for the image but uses the one that was initially used by the application for first batch job. I have not been able to confirm through any documentation so would anyone know if this is indeed the case? I know I should use immutable tags and will look into this shortly.

asked a year ago135 views
1 Answer
0

You are correct in your observation. EMR Serverless does not automatically refresh or check for updates to images stored in Amazon ECR, even if you're using a single tag for your images.

When you create an EMR Serverless application with a custom image, it uses the image specified at the time of creation or last update. The application will continue to use this same image for subsequent job runs, regardless of whether the image in ECR has been updated.

To use a new version of your custom image, you need to explicitly update your EMR Serverless application. This involves specifying the new image URI when updating the application configuration.

Your approach of using a single tag for your images is not ideal for version control and reproducibility. As you've mentioned, it's a good practice to use immutable tags or image digests. This ensures that each version of your application is tied to a specific, unchanging image version.

When you're ready to update your application to use a new image version, you'll need to:

  1. Push the new image version to ECR with a new tag or digest.
  2. Update your EMR Serverless application configuration with the new image URI.

This manual update process gives you more control over when your application starts using a new image version, allowing for proper testing and validation before deployment.

Remember to ensure that your custom images are built on the correct base image for your EMR version, don't modify critical environment variables, and follow other best practices for EMR Serverless custom images.
Sources
ImageConfigurationInput - Amazon EMR Serverless
Customizing an EMR Serverless image - Amazon EMR

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.