Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen
AWS re:Postre:Post

MWAA (2.2.2) Snowflake Connection

0

I am having trouble connecting MWAA to snowflake. I used the MWAA UI to automatically create a VPC, security group, and IAM for my airflow environment. I can not get snowflake to show as an option in the connector, even with the requirements and constraints doc set up.

I tried following this tutorial: https://techholding.co/blog/airflow-mwaa-integrations-pt1/

web server logs indicated that requirements.txt ran successfully

--constraint "/usr/local/airflow/dags/constraints-3.7-mod.txt"

apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-snowflake==3.0.0
snowflake-connector-python==2.7.8
snowflake-sqlalchemy==1.3.4

Since the VPC, security group, and IAM roles are all the defaults that MWAA auto-generates is it possible that they were not configured properly for external connections (services outside the AWS suite, like snowflake)?

  • rePost-User-4042198
    vor einem Jahr

    I now have the Connection Type showing Snowflake (yay!!), but when I put in all my credentials I get this error:

    250001: 250001: Could not connect to Snowflake backend after 0 attempt(s).Aborting

    The MWAA environment has web server access set to Public network and the new connection has the following configuration:

    Connection Type: Snowflake
    Host: https://abc1234.us-east-2.aws.snowflakecomputing.com
    Schema: PUBLIC
    Login: my_login_name
    Password: my_password
    Account: abc1234
    Warehouse: COMPUTE_WH
    Database: RAW
    Region: us-east-2
    Role: ACCOUNTADMIN

    Any idea what could be causing the connection to fail? I read in stackoverflow that removing the 'https://' from the host input helps but that did not seem to do anything when I tested myself.

Themen
Vernetzung & Bereitstellung von InhaltenAnalysenSpeicherAnwendungsintegrationSicherheit, Identität & Konformität
Tags
Amazon VPCData LakesAmazon verwaltete Workflows für Apache Airflow (MWAA)IAM-Richtlinien
Sprache
English
rePost-User-4042198
gefragt vor einem Jahr1462 Aufrufe
1 Antwort
1

If you have a private webserver it has no network egress, and as such you must include the additional Python libraries as WHL files:

https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels

Alternately, you can specify in URI format in Secrets Manager or SSM:

https://docs.aws.amazon.com/mwaa/latest/userguide/connections-secrets-manager.html

If using a public webserver, check your requirements install webserver logs in CloudWatch for errors.

AWS
John_J
beantwortet vor einem Jahr
  • rePost-User-4042198
    vor einem Jahr

    Thanks John, I set the Web Server Access to public network. From the web server logs I figured out that there was a conflict between the requirements file and the constraints file. I removed the versions from the requirements file and literally everything from inside the constraints file (but still left the empty file there) and tried again. That seems to have worked. I can now get snowflake to show as an option in the connection type drop down.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen

Relevanter Inhalt