Requirements installation timeout

1

We recently updated our Airflow environment to version 2.6.3.
After few days we wanted to update the plugins without changing the requirements, but started getting an error
WARNING: requirements.txt installation timed out after 9 minutes. Some requirements may not have installed
It is specified as a warning, but is actually an error, as it interrupts the update.
All out requirements are delivered as wheels (only .whl files, not sdist) inside plugins.zip and their versions are pinned using constraints.txt.

How to solve this problem? It never occurred before, but now it happens all the time.

已提问 8 个月前262 查看次数
1 回答
0

Hi,

The error "requirements.txt installation timed out after 9 minutes" occurs because the environment was spending time for backtracking to find a compatible versions. For this, pip will try every possible combination that it needs to and determine that there is no compatible set. It looks like it could not find it in given time frame[1]. This is also mentioned in the AWS documentation [2] - "Although Amazon MWAA doesn't limit the size of a plugins.zip file explicitly, if dependencies can't be installed within ten minutes, the Fargate service will time-out and attempt to rollback the environment to a stable state".

In your case, I suspect if there is any package version incompatibility introduced in the environment due to airflow version difference that might be creating difficulties for pip to install the packages. I recommend you to perform a testing using the MWAA local runner [3] using the same plugins and requirements to find out the culprit dependency and try to either eliminate it or change the package version. Also, if you plan on changing any package version, make sure that it aligns with the latest constraint file [4].

References:

[1] https://pip.pypa.io/en/latest/topics/dependency-resolution/

[2] https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html

[3] https://github.com/aws/aws-mwaa-local-runner

[4] https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.10.txt

AWS
支持工程师
已回答 8 个月前
profile picture
专家
已审核 2 个月前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则

相关内容