how to avoid the SIGKILL signal?
Despite the optimized DAG and chunk loading, sometimes tasks are killed automatically, turning into SIGKILL status. Googling this error, most often you can see that it occurs from lack of memory (although, I still delete variables and clear memory during script execution. Question: how to avoid the SIGKILL signal? How to understand whether it is necessary to increase the Environment Airflow MWAA or not, And if increased, by how much?
- 新しい順
- 投票が多い順
- コメントが多い順
Hi, I agree: SIGKILL with Airflow often results from insufficient resources (compute, memory, etc.).
To understand how much your DAG consumes, you can use the CLoudWatch metrics provided by MWAA: see section "Apache Airflow metrics available in CloudWatch" in https://docs.aws.amazon.com/mwaa/latest/userguide/access-metrics-cw-202.html
You will then understand where you consume most and be able to adapt your setup accordingly to avoid the SIGKILLs
Best,
Didier
