ECS/Fargate - EBS volume mount has incorrect (root) ownership/permissions.

Question

We are having issues with the relatively recent feature for [mounting EBS volumes to ECS/Fargate tasks](https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ecs-fargate-integrate-ebs/).

The problem is that the mounted volume (folder) is owned by *root*, not the application user (*cloud-web* in this case).

We have an ECS service deployed via Terraform that mounts one EBS volume to a specific path.
The gist of the terraform code is this:

```
# the ecs service
resource "aws_ecs_service" "cloud-web" {
  task_definition = aws_ecs_task_definition.cloud_td.arn
...
  volume_configuration {
    name = "cloud-ebs-volume"
    managed_ebs_volume {
      role_arn    = aws_iam_role.ecs_infrastructure_role.arn
      size_in_gb  = 50
      volume_type = "gp3"
      iops        = 3000
      throughput  = 125
      encrypted   = true
    }
  }

resource "aws_ecs_task_definition" "cloud_td" {
...
  container_definitions = jsonencode([
    {
      image     = "${data.aws_caller_identity.current.account_id}.dkr.ecr.${var.default_region}.amazonaws.com/my-app:latest"
      mountPoints = [
        {
          sourceVolume  = "cloud-ebs-volume"
          containerPath = "/home/cloud-web/data/"
          readOnly      = false
        }
      ]
...
```

This is the relevant portion of our Dockerfile:

```
FROM eclipse-temurin:24-jre-alpine

...

# create user
ARG UID=10001
# NOTE: home dir /home/cloud-web is created by default
RUN adduser --uid ${UID} --disabled-password cloud-web

USER ${UID}
WORKDIR /home/cloud-web
RUN mkdir -p data/results

ENTRYPOINT ["/app/entrypoint.sh"]
```

The app starts fine but the problem is when it tries to write in the /home/cloud-web/data/results/ directory.
It gets "Permission denied" because the /home/cloud-web/data/ folder is owned by root; also there's no 'results' subfolder.

It seems to me that the volume is mounted (to /home/cloud-web/data folder) just before the ENTRYPOINT is executed - if that was the only thing that happened it would be fine (the app can create the 'results' subfolder itself).
However, the problem is that the 'data' folder is owned by root - that's unexpected because it's inside the /home/cloud-web folder
which is created when the 'cloud-web' user is added in the docker image (see "RUN adduser ...").

When using standard docker volumes this never happens and the /home/cloud-web/data folder always has the expected ownership (that is 'cloud-web' is the owner, not 'root').

Based on the information in [Docker volume permissions in ECS](https://serverfault.com/questions/991032/docker-volume-permissions-in-ecs),
it seems that EBS volumes mounted inside Fargate containers act as if they were "bind mounts" and therefore they are owned by 'root'.

Are my observations correct? 
What's the right way to overcome this issue?

NOTE: adding `RUN chown -R cloud-web:cloud-web /home/cloud-web` later in Dockerfile doesn't help - 
it's always shadowed by the EBS volume mounted to /home/cloud-web/data.

RESOURCES
- https://stackoverflow.com/questions/69308389/docker-is-volume-mounted-before-running-cmd
- https://serverfault.com/questions/984578/change-permissions-for-named-volumes-in-docker
- https://serverfault.com/questions/991032/docker-volume-permissions-in-ecs
- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ebs-volumes.html

re:Post Agent · Accepted Answer

Your observations are correct. When using EBS volumes with ECS/Fargate tasks, the mounted volumes are owned by root, similar to how bind mounts work in Docker. This is different from standard Docker volumes where the ownership would match the user that created the directory in the image.

According to AWS documentation, containers in tasks must run as root users to write to mounted Amazon EBS volumes. This is a limitation of how EBS volumes are attached to Fargate tasks.

To overcome this issue, you have a few options:

1. Run your container as root: This is the simplest solution but may not be ideal from a security perspective. You would need to remove the `USER ${UID}` directive from your Dockerfile.

2. Use an entrypoint script that changes permissions: Create an entrypoint script that runs as root, changes the ownership of the mounted directory, and then switches to your application user. This requires using the `gosu` or similar tool:

```
#!/bin/sh
# This would be your entrypoint.sh
chown -R cloud-web:cloud-web /home/cloud-web/data
exec gosu cloud-web /your-actual-application
```

3. Use a sidecar container: Deploy a sidecar container in the same task that mounts the same volume and is responsible for setting up permissions before your application container starts.

4. Use a volume initialization container: Similar to the sidecar approach, but use an initialization container that runs before your application container and sets up the permissions.

Option 2 is likely the most straightforward solution for your case, but it requires your entrypoint script to run as root initially.

Remember that these workarounds are necessary because of how EBS volumes are attached to Fargate tasks, which differs from standard Docker volume behavior.  
**Sources**  
[Use Amazon EBS volumes with Amazon ECS - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ebs-volumes.html)

ECS/Fargate - EBS volume mount has incorrect (root) ownership/permissions.

Add your answer

Relevant content