By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

MWAA (2.2.2) Snowflake Connection

0

I am having trouble connecting MWAA to snowflake. I used the MWAA UI to automatically create a VPC, security group, and IAM for my airflow environment. I can not get snowflake to show as an option in the connector, even with the requirements and constraints doc set up.

I tried following this tutorial: https://techholding.co/blog/airflow-mwaa-integrations-pt1/

web server logs indicated that requirements.txt ran successfully

--constraint "/usr/local/airflow/dags/constraints-3.7-mod.txt"

apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-snowflake==3.0.0
snowflake-connector-python==2.7.8
snowflake-sqlalchemy==1.3.4

Since the VPC, security group, and IAM roles are all the defaults that MWAA auto-generates is it possible that they were not configured properly for external connections (services outside the AWS suite, like snowflake)?

  • rePost-User-4042198
    a year ago

    I now have the Connection Type showing Snowflake (yay!!), but when I put in all my credentials I get this error:

    250001: 250001: Could not connect to Snowflake backend after 0 attempt(s).Aborting

    The MWAA environment has web server access set to Public network and the new connection has the following configuration:

    Connection Type: Snowflake
    Host: https://abc1234.us-east-2.aws.snowflakecomputing.com
    Schema: PUBLIC
    Login: my_login_name
    Password: my_password
    Account: abc1234
    Warehouse: COMPUTE_WH
    Database: RAW
    Region: us-east-2
    Role: ACCOUNTADMIN

    Any idea what could be causing the connection to fail? I read in stackoverflow that removing the 'https://' from the host input helps but that did not seem to do anything when I tested myself.

Topics
Networking & Content DeliveryAnalyticsStorageApplication IntegrationSecurity, Identity, & Compliance
Tags
Amazon VPCData LakesAmazon Managed Workflows for Apache Airflow (MWAA)IAM Policies
Language
English
rePost-User-4042198
asked a year ago1434 views
1 Answer
1

If you have a private webserver it has no network egress, and as such you must include the additional Python libraries as WHL files:

https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels

Alternately, you can specify in URI format in Secrets Manager or SSM:

https://docs.aws.amazon.com/mwaa/latest/userguide/connections-secrets-manager.html

If using a public webserver, check your requirements install webserver logs in CloudWatch for errors.

AWS
John_J
answered a year ago
  • rePost-User-4042198
    a year ago

    Thanks John, I set the Web Server Access to public network. From the web server logs I figured out that there was a conflict between the requirements file and the constraints file. I removed the versions from the requirements file and literally everything from inside the constraints file (but still left the empty file there) and tried again. That seems to have worked. I can now get snowflake to show as an option in the connection type drop down.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content