Not enough space to install requirements?

Question

My MWAA environment will not properly initialize because the requirements install fails. There is nothing in the documentation about a download size limit and no way to override this. There is only a 'recommendation' about size because it can cause slowness in updating.

Error log:

`2023-05-09T16:13:25.377-04:00      Downloading tensorflow-2.8.2-cp37-cp37m-manylinux2010_x86_64.whl (497.9 MB)
2023-05-09T16:13:27.753-04:00      ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device
`

Answer

Per https://github.com/pypa/pip/issues/5816#issuecomment-425410189 you may be able to use `--no-cache-dir` by installing via startup script, i.e.:

```
#!/bin/sh

if [[ "${MWAA_AIRFLOW_COMPONENT}" != "webserver" ]]
then
     # place requirements.txt in DAGs folder, and replace existing requirements.txt with an empty text file
     pip3 install --no-cache-dir -r /usr/local/airflow/dags/requirements.txt
fi
```

See https://docs.aws.amazon.com/mwaa/latest/userguide/using-startup-script.html#create-startup-script

You may also install the files from WHL per https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels

Not enough space to install requirements?

相關內容