How do I resolve errors that I receive when I use the Apache Airflow CLI in an MWAA environment?

Short description

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Amazon MWAA allows users to use Apache Airflow CLI commands to interact with their Apache Airflow environment. These commands are processed by the web server of an Amazon MWAA environment and use the Apache Airflow CLI token for authentication.

Note: It's a best practice to use only the Apache Airflow CLI for synchronous actions such one-time operations. It's not a best practice to use the Apache Airflow CLI for asynchronous actions such as invoking your DAGs. To preserve web server availability for all users, limit the usage of Apache Airflow commands to a maximum of 4 concurrent commands.

The following are common errors that you might receive when you use the Apache Airflow CLI in your Amazon MWAA environment:

• Access denied errors
• DAG could not be found error
• 4## HTTP response code errors
• 5## HTTP response code errors

Resolution

Resolve the error messages that you receive when you use the Apache Airflow CLI based on the following types of errors:

Access denied errors

Check if you received the following error message or similar:

"AccessDeniedException: An error occurred (AccessDeniedException) when calling the CreateCliToken operation: User: is not authorized to perform: airflow:CreateCliToken on resource."

To resolve this error, make sure that the configured AWS Identity and Access Management (IAM) principal has the required permissions policy statement. The permissions policy statement must allow the IAM principal to perform the airflow:CreateCliToken API. Attach the following IAM policy to your IAM principal to allow the Apache Airflow CLI token to be created:

Note: Replace example-region with the required AWS Region, example-account-number with the required account number, and example-environment-name with the name of the environment.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "airflow:CreateCliToken"
      ],
      "Resource": "arn:aws:airflow:::environment/"
    }
  ]
}

DAG could not be found error

Apache Airflow CLI commands such as dags list and dags next-execution require the DAG file to be parsed. If you have a DAG that's processed by a package that isn't installed on the web server, then the following error or similar appears:

"airflow.exceptions.AirflowException: Dag 'sample\_\_dag' could not be found; either it does not exist or it failed to parse."

For Amazon MWAA environments that use a private only web server, the web server doesn't have access to the internet to install additional packages. The additional packages can be found on the Python Package Index public repository. For more information, see Python Package Index on the Python Package Index website. If the web server attempts to install packages from the Python Package Index or another publicly hosted repository, then the package installations fail. To resolve this issue, install packages listed in the requirement file from Python wheels (.whl) files.

Note: After the additional packages are installed on the web server, the Apache Airflow CLI commands that require the DAG file to be parsed run successfully.

4## HTTP response error codes

400 HTTP response - Command parsing error

This error indicates an Apache Airflow command parsing error. This error occurs when an Apache Airflow CLI command is submitted with incorrect syntax such as a missing parameter or invalid option. To resolve this error, make sure that you use the correct syntax for the Apache Airflow CLI command that you use. For more information, see Command Line Interface and Environment Variables Reference on the Apache Airflow website.

Note: For open source Apache Airflow CLI commands, the command structure airflow [-h] GROUP_OR_COMMAND is used. For Apache Airflow CLI commands in the Amazon MWAA environment, the command structure [-h] GROUP_OR_COMMAND is used.

403 forbidden HTTP response - You don't have permission to access the requested resource

This error indicates that you don't have the permission to access the requested resource and that it's either read-protected or not readable by the server. This error occurs when an Apache Airflow CLI command is submitted with an expired authentication token. The authentication token is valid for 60 seconds and is a hard limit that can't be adjusted or increased. To resolve this error, generate a new token and submit the Apache Airflow CLI command again.

Note: Amazon MWAA uses two tokens for authentication: the CLI token and the web login token. The CLI token authenticates users and allows users to submit Apache Airflow CLI commands. The web login token authenticates users that sign in to the Apache Airflow UI for an environment. If an Apache Airflow CLI request uses a web login token, then the token isn't valid and the web server returns the 403 forbidden HTTP response. To resolve this issue, update your script to create a CLI token for your Amazon MWAA environment.

405 HTTP response - Invalid request was made

This error indicates that the request was not valid. This error occurs when you don't use a POST request to submit an Apache Airflow CLI command to the web server. To resolve this error, update your script to use the POST method to make requests to the web server.

405 HTTP response - Oops! Something bad has happened

This error indicates that an unsupported Apache Airflow CLI command was submitted. To resolve this, check that the command you use is supported for the Apache Airflow version that's used in your environment.

5## HTTP response error codes

500 HTTP response - Response Timeout

Apache Airflow CLI commands are processed by the web server in an Amazon MWAA environment. If your environment is configured for private only access, then the web server can't be routed publicly. To resolve this error, provide an available route from the local machine that Apache Airflow CLI commands are submitted from to the private web server. For more information, see How do I access the Apache Airflow UI using the private network access mode in my Amazon MWAA environment?

503 HTTP response

This error indicates that you've exceeded the recommended Apache Airflow CLI command concurrency. This causes your web server resource utilization to increase and might lead to an unavailable web server. To resolve this error, reduce the rate that you submit Apache Airflow CLI commands. Make sure that previous commands are completed before you submit new commands.