AWS DevOps Agent - Troubleshooting Amazon MWAA pipeline failures

When an Airflow pipeline breaks on Amazon Managed Workflows for Apache Airflow (Amazon MWAA), the evidence is almost always sitting in CloudWatch. The hard part is knowing where to look. MWAA ships five Airflow log groups to Amazon CloudWatch Logs (DAGProcessing, Task, Scheduler, Worker, and WebServer), and the on-call engineer still has to figure out which log group, which DAG, and which task actually failed.

This article shows how AWS DevOps Agent takes over that correlation work. It reads the alarm that fired, pulls the metric behind it, reads the relevant MWAA log groups, and returns a root cause with a mitigation plan. I will walk through three common Airflow failure modes, show what the agent concludes for each, and show how the fix gets applied and confirmed.

One thing worth calling out front. MWAA does not publish a per-task-failure or a per-DAG-parse-error metric in the AWS/MWAA namespace. It publishes queues, RDS, and container metrics such as ApproximateAgeOfOldestTask. To get a clean, fault-specific signal you can alarm on, this setup adds CloudWatch log metric filters that scan the log groups and publish counts (ImportErrors, TaskErrors) to a custom MWAADemo/Faults namespace. The alarms then watch those counts.

Architecture

Figure 1. MWAA + AWS DevOps Agent autonomous root-cause analysis flow.

The developer deploys the stack and uploads DAGs to Amazon S3, and MWAA syncs and runs them. When a fault is injected, MWAA streams the failure to CloudWatch Logs, the log metric filters count it into the MWAADemo/Faults namespace, and a CloudWatch alarm breach. AWS DevOps Agent picks up the alarm, correlates the metrics and logs, and returns a root cause with a mitigation plan.

Environment used in this article

• Apache Airflow 2.10.3 on Amazon MWAA (mw1.small), single region.
• A healthy pipeline (healthy_demo_pipeline) running green on a 15-minute schedule. This is a steady state.
• Two CloudWatch alarms for the agent to investigate:

Solution overview

• Broken DAG (import error). A DAG imports a package that is not installed, so it fails to parse time on every parse cycle. This is the standing incident. Fix: add the package to requirements.txt or remove the import, then update the environment.
• Task runtime exception. A single task raises an error and turns red. Fix: add error handling and a retry with backoff around the failing call and make the task idempotent.
• Retry exhaustion. A task fails every attempt, which points to a persistent outage rather than a transient blip. Fix: treat it as an infrastructure problem and restore the dependency. More entries will not help.

The agent does more than name the cause. It produces a prioritized mitigation plan, posts it back through its ticketing or chat integrations (or hands the change to Kiro), and you confirm closure when the alarm goes back to OK. The section "From root cause to applied fix" covers that loop. The article then lists the CloudWatch signals worth watching day to day, and a few things to keep in mind if you run this pattern in production.

Troubleshooting the Airflow failure modes

Failure mode 1: Broken DAG (import error)

Scenario. The Airflow UI shows a "Broken DAG" banner, and the <env>-DagImportErrors alarm goes to ALARM and stays there. A broken DAG is re-parsed on every scheduler cycle, so the alarm keeps re-firing. That makes it the easiest incident to demo against, because it does not clear on its own, and you can start the investigation whenever you like.

Cause. A DAG imports a Python dependency that is not in the environment's requirements.txt, for example import pandas_supercharge. Airflow cannot import the module, so the DAG never registers and never runs. The failure lands in the DAGProcessing log group as a traceback:

# CloudWatch Logs -> airflow-<env>-DAGProcessing
 Traceback (most recent call last):
   File "/usr/local/airflow/dags/fault_import_error_pipeline.py", line 24, in <module>
     import pandas_supercharge
 ModuleNotFoundError: No module named 'pandas_supercharge'
 Broken DAG: [/usr/local/airflow/dags/fault_import_error_pipeline.py]

How AWS DevOps Agent diagnoses it. The agent reads the <env>-DagImportErrors alarm, follows the ImportErrors metric back to the airflow-<env>-DAGProcessing log group, and surfaces the ModuleNotFoundError. It should conclude that the DAG imports the uninstalled package pandas_supercharge, which is what breaks the DAG.

Fix. Add the missing package to requirements.txt and update the MWAA environment's requirements version, or remove the import from the DAG. Once the DAG parses cleanly in the UI, the alarm returns to OK.

Verify. In the investigation chat, ask: "What is the root cause of the <env>-DagImportErrors alarm? Which DAG file and which import failed?" A good answer names the DAG file and the missing module.

Failure mode 2: Task runtime exception

Scenario. One task turns red (FAILED) in the grid view, and the <env>-TaskFailures alarm goes to ALARM for a short window. A single failed task is transient, so the alarm clears once the run finishes.

Cause. A task raises an unhandled exception at run time. In the demo, call_downstream_api raises a RuntimeError because a downstream dependency returned HTTP 503, and there is no error handling or fallback.

How AWS DevOps Agent diagnoses it. The agent ties the TaskErrors metric to the airflow-<env>-Task log group, finds the failing task instance, and reads the traceback. It should call out the failing task, the exception type, and the HTTP 503 behind it.

Fix. Wrap the external call in error handling with a retry and backoff, and make the task idempotent so a re-run is safe. If a single downstream blip should not take down the whole pipeline, add a fallback or circuit breaker.

Verify. Ask: "Focus on log group airflow-<env>-Task and identify the failing task and exception." The agent should return the task name and the RuntimeError and HTTP 503 detail.

Failure mode 3: Retry exhaustion (persistent outage)

Scenario. The task shows four failed attempts and then FAILED, and <env>-TaskFailures stays in ALARM across the retries.

Cause. The task fails on every attempt. In the demo, connect_to_db uses up all four retries against db-prod-cluster:5432. Because every attempt fails the same way, this is a persistent connectivity outage, not a flaky call.

How AWS DevOps Agent diagnoses it. The agent reads the repeated ConnectionError entries across all attempts in the Task log group and reasons about the pattern. Identical failures on every retry point to a dependency that is down, not to a transient error. That distinction is the whole point of this case, and it is what the agent should state.

Fix. Handle it as an infrastructure incident rather than a code retry problem. Check the database or endpoint health, the security groups, and the network path from the MWAA workers. Retrying more times will not help while the dependency is down.

Verify. Ask: "The task failed 4 times, is this a transient issue or a persistent outage? Justify from the logs." A good answer cites the identical failure on every attempt as evidence that the dependency is down.

From root cause to applied fix

Finding the cause is only half the job. Once AWS DevOps Agent has the root cause, it produces a mitigation plan and pushes it toward resolution through the same integrations it used to investigate.

It writes the fix back to where the incident lives. The root cause and plan go into the originating ticket or chat, such as a ServiceNow or Jira incident or a Slack channel, so the on-call engineer gets something actionable instead of a raw alarm.

For code and config fixes, which is most of what you hit with Airflow, it can hand the change to Kiro. The plan becomes a concrete change: edit requirements.txt, drop a bad import, or add a retry with backoff. Kiro can apply that change and open a pull request, so a human still reviews and merges it.

And because every fault maps to a CloudWatch alarm, "fixed" is not a matter of opinion. The alarm returns to OK and stays there, and you verify closure against the same signal that opened the incident.

Here is how that plays out for the three modes:

Failure mode	Agent's root cause	Applied fix	Confirmed by
Broken DAG (import)	DAG imports uninstalled pandas_supercharge	Add package to requirements.txt or remove the import, then update the requirements version	<env>-DagImportErrors back to OK, DAG parses in the UI
Task runtime exception	call_downstream_api raised RuntimeError on HTTP 503, no handling	Add error handling, retry with backoff, and idempotency, then redeploy the DAG	<env>-TaskFailures back to OK, task run succeeds
Retry exhaustion	connect_to_db failed all 4 attempts to db-prod-cluster:5432, a persistent outage	Restore the dependency (health, security group, network path), retries alone will not help	<env>-TaskFailures back to OK once the dependency is healthy

The useful part is that the loop closes against the same alarm that opened it. You do not just get a theory, you get a change to apply and a clear signal that tells you the Airflow issue is actually resolved.

Signals worth watching

DAGProcessing log group. Where import and parse errors show up first. Drives the ImportErrors metric.

Task log group. Task tracebacks and retries. Drives the TaskErrors metric.

Scheduler log group. Scheduling decisions and DAG run state.

• Custom MWAADemo/Faults namespace. The fault-specific counts that make each failure something you can alarm on.

A few things to keep in mind

• MWAA has no native per-failure metrics. To alarm reliably on a broken DAG or a failed task, publish your own counts from CloudWatch log metric filters. If you rely only on AWS/MWAA metrics, you will miss these failures.
• A broken DAG is a standing incident. Import errors recur on every parse cycle, so the alarm stays in ALARM until you fix the DAG. A single task failure, by contrast, is transient. That is why the import case is the most reliable one to demo an investigation against.
• Retry count is a signal, not just a symptom. Identical failures across every retry mean the dependency is down, and that changes the fix. Do not read four failed attempts as "just retry more."
• Give the agent a good starting point. Kicking off the investigation from the latest alarm points it straight at the right metric and log group. A vague prompt makes it work harder to find failure.
• Keep topology access at least privilege. The agent uses a read-only role to inspect CloudWatch, MWAA, VPC, and S3. Scope it to what the investigation needs.

Conclusion

Airflow-on-MWAA pipeline failures usually come down to one of three things: a DAG importing a package that is not in requirements.txt, a task raising an unhandled exception, or a task burning through its retries against a dependency that is down.

Because MWAA streams all its Airflow logs to CloudWatch, and because log metric filters turn those logs into metrics, you can alarm on, AWS DevOps Agent can start from a single alarm and work back to the exact DAG, task, and error on its own. From there it produces a mitigation plan, writes it back to the ticket or chat (or hands the change to Kiro), and you confirm the fix when the alarm returns to OK. It is the same autonomous root-cause pattern AWS has shown for other services (see Sources), applied to the Airflow orchestration layer.

To learn more about how AWS Support plans and offerings can help you run AWS DevOps Agent at scale, see AWS Support.

Sources

• AWS DevOps Agent User Guide, About AWS DevOps Agent
• AWS DevOps Agent User Guide, Autonomous incident response
• AWS DevOps Agent User Guide, Incident Response
• Amazon MWAA User Guide, Monitoring dashboards and alarms
• Amazon MWAA User Guide, Accessing Airflow logs in CloudWatch
• Amazon MWAA User Guide, Troubleshooting: DAGs, Operators, Connections
• AWS Blog, Autonomous Root Cause Analysis for AWS Systems Manager Patch Failures Using AWS DevOps Agent
• AWS Blog, Automated network incident response with AWS DevOps Agent
• AWS Blog, Automating CloudWatch dashboards and alarms for Amazon MWAA