By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Does MWAA Airflow v2.2.2 support custom XCom backends?

0

Description

While trying to register a custom XCom backend in MWAA v2.2.2 I'm getting an INCORRECT_CONFIGURATION on the cluster console UI.

Steps to reproduce

  • In my MWAA local environment, my custom XCom backend class lives in the plugins folder and works perfectly.
  • In airflow.cfg file I have it registered as follows:
[core]
...
xcom_backend = xcom.s3_backend.S3XComBackend
  • In order to register the XCom backend in MWAA I provide this Airflow configuration option (on MWAA cluster Edit UI):
core.xcom_backend: xcom.s3_backend.S3XComBackend
  • MWAA cluster update fails with the INCORRECT_CONFIGURATION message.
  • I also tried moving the custom XCom module to the dags folder with the same result.

Question

Am I doing something wrong or are custom XCom backends currently not supported for MWAA? Thank you!

Topics
Application Integration
Tags
Amazon Managed Workflows for Apache Airflow (MWAA)
Language
English
Kermit
asked 2 years ago691 views
2 Answers
0

MWAA does not explicitly block custom Xcom backends provided that the execution role and network both have access, however using custom Xcom backends may diminish performance as every task will have to write to that backend. Further, if using a private web server, there may not be network egress to the backend store.

AWS
John_J
answered 2 years ago
  • Kermit
    2 years ago

    Thank you for your answer! So, assuming that MWAA allows using custom backends, I suspect that there must be something wrong with custom code visibility, because when I set the configuration option like this

    core.xcom_backend -> airflow.models.xcom.BaseXCom

    MWAA cluster restarts successfully but when I change airflow.models.xcom.BaseXCom to my custom path (to either dags or plugins) - xcom.s3_backend.S3XComBackend, it crashes on restart.

    The module is stored at plugins/xcom/s3_backend.py, the plugins folder is zipped and put into the Airflow bucket, the rest of the plugins are seen by MWAA.

    Is the above method a proper way of setting up a custom backend? And maybe there's a way of looking up the detailed error behind the INCORRECT_CONFIGURATION message?

0

anything new ?

call_me_terry
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content