Secrets backend support in MWAA

2

Is any other secrets backend besides AWS Secrets manager supported in MWAA? I would like to use Hashicorp Vault, and the Airflow provider loads fine, but when i try to set secrets.backend_kwargs it seems fail.

BenO
gefragt vor einem Jahr688 Aufrufe
2 Antworten
1

Facing same issue, when I set Vault as secret backend MWAA env. stuck for couple of hours in create/update state and then fails after couple of hours

Using AWS Secrets manager works fine

Also, manually initiating VaultBackend class in Python operator to get secrets works secrets fine

So, I assume there is no connectivity issue in my case

It is very complicated to analyse such failure, as there is no log and Error message is pointing to documentation

Grisha
beantwortet vor einem Jahr
-1

Hello, As you may already aware, Amazon MWAA is a managed service for Apache Airflow that lets you use your current, familiar Apache Airflow platform to orchestrate your workflows. MWAA runs open source Apache Airflow. So if Airflow supports a third party integration, then usually you can be able to use it with MWAA.

Airflow supports Hashicorp Vault as shown here. https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/secrets-backends/hashicorp-vault.html. The possible reasons for this failure could be IAM permissions issues or networking issues. If the vault is stored externally then please ensure whether you are able to connect it from the subnet you are using in MWAA. Please make sure to have a route to connect to vault if is placed inside EC2.

Usage of a custom secrets backend is outside of the scope of the MWAA service team, and in general we cannot provide specific guidance. Note that the secrets backend may need to be accessible from the web server, depending on how it's implemented, and web servers in MWAA cannot access VPCs outside AWS.

AWS
beantwortet vor einem Jahr
  • One follow on question, when the MWAA is in Private network mode, is it possible for the web servers to initiate traffic back into my VPC? Our vault is in our VPC.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen