how to avoid the SIGKILL signal?

Question

Despite the optimized DAG and chunk loading, sometimes tasks are killed automatically, turning into SIGKILL status. Googling this error, most often you can see that it occurs from lack of memory (although, I still delete variables and clear memory during script execution. Question: how to avoid the SIGKILL signal? How to understand whether it is necessary to increase the Environment Airflow MWAA or not, And if increased, by how much?

Accepted Answer

Hi, I agree: SIGKILL with Airflow often results from insufficient resources (compute, memory, etc.).

To understand how much your DAG consumes, you can use the CLoudWatch metrics provided by MWAA: see section "Apache Airflow metrics available in CloudWatch"  in https://docs.aws.amazon.com/mwaa/latest/userguide/access-metrics-cw-202.html

You will then understand where you consume most and be able to adapt your setup accordingly to avoid the SIGKILLs

Best,

Didier

how to avoid the SIGKILL signal?

Relevant content