Skip to content

MWAA worker fleet stops running any tasks — MinWorkers not honored, FORCED worker-replacement updates report SUCCESS but provision no worker

0

We have an MWAA environment (mw1.medium, MinWorkers=1, MaxWorkers=10, us-east-2) where the Celery worker fleet has stopped running any tasks. Environment.Status reports AVAILABLE the whole time, so nothing in the environment's own status surfaces the problem.

Evidence:

  1. No worker container has logged a single line in ~46+ hours (and counting):
aws logs describe-log-streams --log-group-name airflow-<env>-Worker \
   --order-by LastEventTime --descending --max-items 1 \
   --query "logStreams[0].{lastWorkerLogEver:lastEventTimestamp,stream:logStreamName}"

Returns a lastEventTimestamp far in the past, never advancing on re-checks.

  1. No new worker instance has even been provisioned — every fresh worker boot writes a requirements_install_ log* stream, and the newest one of those is ~2 days old.

  2. AWS/MWAA CloudWatch metrics for the environment: RunningTasks = 0 continuously for 6+ hours sampled; QueuedTasks = 1 continuously over the same window — one task permanently queued with nothing ever picking it up.

  3. We ran UpdateEnvironment twice with WorkerReplacementStrategy=FORCED (once implicitly via a RequirementsS3Path change, once explicitly with --worker-replacement-strategy FORCED --min-workers. Both calls returned LastUpdate.Status: SUCCESS — but neither resulted in any new worker container appearing (no new requirements_install_* or worker_* log stream, RunningTasks stayed at 0 throughout).

Since MWAA is fully managed, we have no visibility into the underlying ECS/Fargate task that would show why worker provisioning is silently failing.

Question: Has anyone seen MWAA silently fail to (re)provision workers like this — where MinWorkers isn't honored and a FORCED worker-replacement update reports success without actually launching a worker? Is there a way to force-diagnose or recover this from the customer side without Business support?

3 Answers
0
Accepted Answer

I could solve this issue. It was caused by another MWAA deployment that set up VPC Endpoints and highjacked DNS resolution for the whole VPC. So I set a wider HTTPS SG Rule to stop blocking traffic.

answered a month ago

EXPERT

reviewed 24 days ago

0

What you're experiencing appears to be a critical provisioning failure in your MWAA environment where the control plane is reporting AVAILABLE and update operations are returning SUCCESS, but the worker fleet is not actually being provisioned. This is particularly concerning because forced updates should immediately stop all active workers and provision new ones, yet you're seeing no new worker containers at all.

Unfortunately, the sources available don't directly address this specific scenario where the MWAA control plane silently fails to provision workers despite reporting success. The documentation covers worker replacement strategies and mentions that forced updates take approximately twenty to thirty minutes, but doesn't cover cases where the provisioning itself fails without surfacing an error.

From a troubleshooting perspective, here are some areas to investigate:

First, verify that your environment's underlying resources are intact - ensure your S3 bucket is accessible, has versioning enabled, and that the VPC configuration (security groups, subnets, route tables) hasn't changed. MWAA workers need proper network connectivity to provision successfully.

Second, check if there are any IAM permission issues that might prevent worker provisioning. While the environment shows AVAILABLE, underlying permission problems could prevent ECS task launches without surfacing in the environment status.

Third, examine whether there are any VPC resource constraints - such as insufficient IP addresses in your subnets or security group rule limits - that could silently block worker provisioning.

Given that this is a managed service failure where the control plane is misreporting the actual state of your worker fleet, and you've already attempted forced updates without success, you should open a support case through the AWS Support Center at https://console.aws.amazon.com/support/. This appears to be a service-level issue that requires AWS to investigate the underlying ECS/Fargate provisioning layer, which isn't accessible to customers. Include all the diagnostic evidence you've gathered - the CloudWatch metrics showing zero running tasks, the log stream timestamps, and the update operation results - as this will help expedite the investigation.
Sources
Update an Amazon MWAA environment - Amazon Managed Workflows for Apache Airflow

answered a month ago

  • Ruled out the suggested causes: subnets have thousands of free IPs, security groups are all intact and unchanged, IAM execution role is unchanged with no permissions boundary. Bucket versioning is off, but that affects DAG-file tracking, not worker provisioning.

    Found something more concrete in CloudTrail: ec2:CreateNetworkInterface calls invoked by airflow.amazonaws.com via AWSServiceRoleForAmazonMWAA — the MWAA control plane itself attempting to launch a container — recurring on a near-exact ~68-minute cadence since this morning's update:

    07:29, 08:36, 09:44, 10:52, 12:00, 13:08, 14:16 (+67-68m each)

    Each attempt creates the ENI successfully (gets a private IP, status pending) but never progresses far enough to write a single CloudWatch log line — not even the requirements_install bootstrap line a fresh worker normally emits first. So MWAA is repeatedly attempting worker provisioning and failing inside its managed Fargate layer, before any customer-visible output.

0

Your CloudTrail finding is the useful part of this thread, and it narrows things considerably. A few observations, then concrete next steps.

What the evidence actually tells you

ENI created successfully → pending → no container output at all, not even the requirements bootstrap line. That means the Fargate task never reached the point of running anything. The failure is in task placement or startup, before the container image executes.

The ~68-minute fixed cadence is consistent with an MWAA-internal reconciliation loop rather than exponential backoff, which fits "the control plane keeps trying and keeps failing at the same step."

Also worth noting: the FORCED update returning SUCCESS isn't necessarily a misreport. SUCCESS on UpdateEnvironment means the configuration change was applied — it doesn't assert that a worker subsequently came up. So that's a weaker signal than it looks.

The one thing I'd check that hasn't been ruled out

You covered subnets, security groups, and IAM. KMS is the notable gap.

aws mwaa get-environment returns the KmsKey in use. If it's a customer-managed key rather than aws/airflow:

aws kms describe-key --key-id <key-arn> \
  --query 'KeyMetadata.{State:KeyState,Enabled:Enabled,Deletion:DeletionDate}'
aws kms get-key-policy --key-id <key-arn> --policy-name default

A CMK that's been disabled, scheduled for deletion, or had its policy tightened will break provisioning while the environment continues reporting AVAILABLE. This is exactly the class of silent failure that produces "no logs at all" — the worker can't decrypt what it needs before it can write anything. Note that AWS's own guidance on MWAA updates flags checking networking, IAM, and KMS before an update for precisely this reason.

Second gap: egress path for image pull. You confirmed security groups are intact, but not the route tables, NAT Gateway, or VPC endpoints. If the NAT Gateway was deleted or its EIP released, the task gets an ENI, fails to pull, and dies before logging. Verify the NAT Gateway is available, its EIP is still attached, and the private subnet route tables still point at it. If you use VPC endpoints instead, check ECR (both api and dkr), S3, Logs, SQS, and KMS.

A way to see into the hidden Fargate layer

This is worth trying. Fargate-managed ENIs carry the ECS attachment identity in their description. Take the ENI IDs from those CloudTrail events:

aws ec2 describe-network-interfaces --network-interface-ids eni- \
  --query 'NetworkInterfaces[0].{Desc:Description,Requester:RequesterId,Status:Status,Groups:Groups}'

The Description field often contains the ECS cluster and attachment ARN, which is more visibility than MWAA normally gives you. Also check whether the ENIs from earlier attempts are accumulating rather than being cleaned up — 46 hours at 68-minute intervals is roughly 40 attempts, and leaked requester-managed ENIs are their own problem.

Recovery paths without Business support

Change the environment class. Moving mw1.medium → mw1.large (and back) forces a full re-provision through a different code path than a FORCED worker replacement. Blunt, but it sometimes shakes loose a stuck internal state.

Rule out the requirements file entirely. Point RequirementsS3Path at an empty or omitted file. If a worker then boots, the problem was dependency installation — though the absence of any requirements_install_* line argues against that.

Create a fresh environment. This is the reliable answer and the one I'd plan for. Same VPC, same execution role, same DAG bucket, new environment name. It costs you the duplicate environment for a while but it's deterministic, whereas waiting on a stuck managed control plane is not. Given you're 46+ hours in with one task permanently queued, I'd start this in parallel rather than sequentially.

Note that subnets can't be changed post-creation, so you can't sidestep a potential single-AZ Fargate capacity problem on the existing environment. A new environment lets you pick different subnets — worth doing if you suspect that.

EXPERT

answered a month ago

  • Thanks Taka.

    • KMS: ruled out. KmsKey on the environment is None — default AWS-managed key, no CMK involved.
    • NAT Gateway / egress: available, EIP still attached. Route tables for both subnets correctly default-route to it.
    • VPC endpoints: all available — S3, Logs, SecretsManager, SQS, plus several PrivateLink service endpoints. No gap there.
    • ENI leak check: not leaking — the ENI from my original CloudTrail evidence is already gone (InvalidNetworkInterfaceID.NotFound), so failed attempts are being cleaned up, not accumulating. At 68-min intervals over 40+ hours that's reassuring, at least.
    • The live-ENI trick: caught one in the act this time (created seconds before I looked) — it actually got to status: in-use / attached, on an instance owned by 480281386609. Unfortunately the Description field is just the random client-token hash, not an ECS task/attachment ARN — so that specific visibility trick doesn't seem to surface anything extra for MWAA's managed Fargate layer specifically. Worth knowing for the next person who tries it.

    On your two recovery suggestions: already tried the environment-class round-trip (mw1.medium → mw1.large → back) — no change, same failure mode, zero worker activity throughout. Agreed on your main point though — going to start standing up a fresh environment in parallel now rather than keep waiting on the stuck one. Will report back if that surfaces anything new.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.