Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Fargate tasks exit after startup

0

Issue Report: Fargate tasks exit ~0.3–0.5s after successful startup, with no error logged

Summary

A FastAPI/Uvicorn container running on Fargate completes application startup successfully (DB migrations run, all config loaded, app logs "ready to accept traffic"), then is stopped roughly 0.3–0.5 seconds later. The container's own logs show a clean shutdown sequence with no exception — even with a global Python exception hook installed, nothing is caught — yet ECS reports the container's exit code as 1 and stop reason "EssentialContainerExited". This reproduces identically across three different deployment configurations.

Environment

  • Account region: us-east-2
  • Cluster: default
  • Launch type: FARGATE, platform version 1.4.0
  • Image: linux/amd64, built and pushed to a private ECR repo
  • Task role and task execution role: separate IAM roles, both with correct trust policies (ecs-tasks.amazonaws.com) and scoped permissions (Secrets Manager GetSecretValue, S3 object access)

Configurations tested (all produce the identical symptom)

  1. ECS Express Mode service, default CANARY deployment strategy, desiredCount = 1
  2. Same Express Mode service, desiredCount = 2 (to rule out canary percentage math on a single-task service)
  3. Standard ECS service (not Express Mode), explicit deploymentController.type=ECS, ROLLING strategy (no canary at all)

All three show the same pattern: task registers with target group (when a load balancer is attached), then drains/deregisters within seconds; task stopCode is EssentialContainerExited, container exitCode is 1, container reason is null.

Evidence the application itself is not crashing

Diagnostic logging was added directly to the app (available on request):

  • Logs every environment variable's presence/absence at startup (all 11 required variables confirmed present in every run)
  • Logs a counter of health-check requests received (/health endpoint) — this counter is 0 in every single run, confirming the load balancer never even gets a chance to reach the container before it's stopped
  • Wrapped the shutdown code path (engine.dispose()) in try/except with full exception logging — no exception is ever logged
  • Installed sys.excepthook to catch and log literally any uncaught exception anywhere in the process — nothing is ever caught

Sample log output from a real run (standard ECS, ROLLING strategy):

INFO - Starting Rent Roll Extractor v1.0.0 [development]
INFO - ENV CHECK — RDS_SECRET_ID: present   (...all 11 present...)
INFO - Database migrations applied successfully
INFO - Corrections store loaded: 5 corrections
INFO - STARTUP COMPLETE — ready to accept traffic
INFO - Rent Roll Extractor shutting down — uptime: 0.5s — received 0 health check request(s)
INFO - SHUTDOWN COMPLETE — engine disposed cleanly

Immediately after this, describe-tasks reports:

{
  "stoppedReason": "Essential container in task exited",
  "stopCode": "EssentialContainerExited",
  "containers": [{ "exitCode": 1, "reason": null }]
}

Additional oddity found while isolating the issue

Attempting to isolate whether this was specific to our image, we tried launching an unrelated public.ecr.aws/nginx/nginx:latest container (via both run-task and create-service) using the same, already-working ecsTaskExecutionRole that our other service uses successfully. Both attempts failed immediately with:

ECS was unable to assume the role 'arn:aws:iam::<account>:role/ecsTaskExecutionRole'
that was provided for this task. Please verify that the role being passed
has the proper trust relationship and permissions and that your IAM user
has permissions to pass this role.

The role's trust policy is confirmed correct (ecs-tasks.amazonaws.com, scoped aws:SourceAccount/aws:SourceArn conditions matching the account and ECS ARN pattern). This role successfully launches tasks for our other service in the same account/region at the same time, so this second issue appears intermittent/inconsistent for new task-definition families.

What we've ruled out

  • IAM permissions (task role and execution role both verified, working for successful DB/Secrets Manager calls when the container briefly runs)
  • Security groups (ALB → task port 80 explicitly allowed, confirmed via direct public-IP curl to the container's /health endpoint succeeding during an earlier debugging session)
  • Container port mismatch (Dockerfile, task definition, target group, and security group all confirmed consistent on port 80)
  • Missing/misconfigured environment variables or secrets (confirmed present via in-app diagnostic logging on every single run)
  • Application-level exceptions (confirmed via try/except and global exception hook — nothing is ever thrown)
  • Express Mode canary-specific behavior (identical symptom on plain ROLLING strategy with Express Mode entirely bypassed)

Ask

  1. Any insight into what would cause Fargate to report exitCode 1 for a container whose own application logs show a clean, exception-free shutdown sequence completing successfully
  2. Any insight into the intermittent ecsTaskExecutionRole assumption failure for new task-definition families despite a correct, working trust policy
  • Florian Turnwald EXPERT
    2 months ago

    If my answer was helpful, I would appreciate it if you could mark it as the accepted answer.

Topics
Containers
Tags
AWS FargateECS Service Discovery
Language
English

asked 2 months ago78 views

2 Answers
2

Adding to the previous answer, while sidecar containers (like GuardDuty or FireLens) are indeed a very common cause for early exits, I noticed a detail in your provided describe-tasks output that might point in a different direction.

The snippet shows "containers": [{ "exitCode": 1, "reason": null }]. Since there is only one object in the array, it suggests this is a single-container task. If a sidecar were failing, we would typically expect to see at least two containers listed in that output. Assuming it is just your application container, the issue might be happening within the container boundary itself. Since your application logs a wonderfully clean shutdown sequence but ECS still reports exitCode 1, I suspect it could be related to one of the following areas:

  • PID 1 and Signal Handling: If you are using a wrapper script (e.g., entrypoint.sh) to start Uvicorn, the script itself might be running as PID 1. When ECS sends a SIGTERM to stop the container, Uvicorn might shut down cleanly (as your logs show), but the wrapper script might subsequently exit with a non-zero code. It might be worth verifying if you are using exec in your entrypoint script to ensure Uvicorn runs as PID 1 and handles the signals directly.

  • ECS Container Health Checks: You confirmed the ALB health checks never reach the container. However, I wonder if there is a native Docker HEALTHCHECK defined in your Dockerfile or an ECS-level Health Check in your Task Definition. If an aggressive health check is configured there and fails immediately during startup, the ECS agent will forcefully stop the container, which often results in a clean app shutdown but a container exit code of 1.

  • Development Mode Flags: I noticed the log states [development]. If Uvicorn is being started with the --reload flag, the underlying file watcher can sometimes behave unpredictably in stateless Fargate environments, which can trigger an unexpected shutdown loop.

Regarding the secondary issue with the Nginx container: I completely agree with the previous assessment. The "unable to assume role" error during run-task is very often not an issue with the Task Execution Role's trust policy, but rather that the IAM User or CI/CD pipeline executing the command lacks the iam:PassRole permission for that specific execution role.

See also:

PS: Hope this gives you a few additional angles to investigate!

EXPERT

answered 2 months ago

EXPERT

reviewed 2 months ago

0

Based on the symptoms you're describing—where your application logs show clean startup and shutdown but ECS reports exit code 1—this pattern is consistent with a GuardDuty Runtime Monitoring sidecar container issue rather than a problem with your application itself.

When GuardDuty Runtime Monitoring is enabled for ECS Fargate, it injects a sidecar container into your tasks. If this sidecar fails to start or exits prematurely (often due to being unable to pull its container image from ECR), it can cause your entire task to be stopped with "EssentialContainerExited" even though your application container is running perfectly. The timing you're seeing (0.3-0.5 seconds after your app starts) aligns with a sidecar initialization failure.

Key troubleshooting steps to investigate:

  1. Check if GuardDuty Runtime Monitoring is enabled: Review your GuardDuty settings in the console to see if ECS runtime monitoring is active for your cluster.

  2. Verify ECR permissions: Your task execution role needs specific ECR permissions to pull the GuardDuty sidecar image. Ensure it has the necessary ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage permissions.

  3. Check network connectivity: The Fargate tasks must be able to reach ECR, either through public internet access or properly configured VPC endpoints. If your tasks are in private subnets without NAT Gateway or VPC endpoints for ECR, the sidecar cannot download.

  4. Review security group configuration: Your security group must allow outbound access to the S3 managed prefix list on port 443, which is required for pulling container images from ECR.

  5. Examine all container logs: Use describe-tasks to check if there are multiple containers in your task definition. If you see a GuardDuty sidecar container, check its specific exit code and reason.

These three components (permissions, network connectivity, and security group configuration) are all necessary for successfully downloading the GuardDuty container image, and a failure in any one will cause the behavior you're experiencing.

Regarding the intermittent IAM role assumption issue with the nginx test: this could be a separate timing or eventual consistency issue, but it's worth verifying that your IAM user or role performing the ECS operations has iam:PassRole permission for the execution role.
Sources
Runtime coverage and troubleshooting for Amazon ECS clusters - Amazon GuardDuty

answered 2 months ago

EXPERT

reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content