Guidance Needed for Migrating Data from Cosmos DB to Snowflake with CDC: AWS Glue

Hello

This third-party documentation indicates that Azure functions can be used to connect to a change feed and trigger on change events. https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/change-feed-functions

You can use the Azure Functions to feed data into an AWS managed streaming service such and Amazon Managed Kafka or Kinesis which Glue natively integrates with Glue Streaming jobs. https://docs.aws.amazon.com/glue/latest/dg/streaming-chapter.html

Glue supports connections to Snowflake using Glue Connections: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-snowflake-home.html

Your cost considerations on the AWS side will be related to your Glue resources used, as well as your data retention and streaming configurations. You cost optimizations with this recommendation with be: Cost Optimizations in Glue: Use native Spark functions, Spark SQL, or Glue Spark functions as much as possible to leverage the Catalyst Optimizer in Spark for efficient jobs. If User Defined Functions are required, where possible use Java User Defined Functions rather than Python User Defined Functions. Cost Optimizations in Streaming: Generally, Kinesis will be cheaper, but the exact pricing will depend on your streaming configurations (e.g.: data retention, etc.)

• Amazon Managed Kafka offers a Pricing Calculator linked in its documentation: https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#brokers-per-cluster
• Kinesis Pricing for Data Streams can be found in its documentation as well for your analysis: https://aws.amazon.com/kinesis/data-streams/pricing/?p=pm&c=aa&pd=kinesis&z=4

Thank you for considering AWS Glue for your ETL needs!