Some Tool to work with Lineage between AWS and SnowFlake

To control the lineage between external sources like Snowflake and AWS services such as AWS Glue and Lambda, you can consider using a metadata management tool or a data catalog. These tools help track the lineage and metadata of your data, providing visibility into its origins and transformations. Here are a few options you can explore:

1. AWS Glue Data Catalog: The AWS Glue Data Catalog is a fully managed metadata repository that integrates with various AWS services. It allows you to define tables, schemas, and metadata for your data sources. By using AWS Glue Crawlers, you can automatically discover the schema and metadata of your data in Snowflake and other sources, and then create a unified metadata catalog that includes lineage information.
2. Apache Atlas: Apache Atlas is an open-source metadata management and data governance tool. It provides capabilities for capturing, organizing, and governing metadata across various data sources, including Snowflake and AWS services. With Apache Atlas, you can define entities, relationships, and lineage information to understand how data flows between systems.
3. Alation: Alation is a popular enterprise data catalog solution that offers metadata management, data lineage, and data governance capabilities. It can integrate with Snowflake and AWS services, allowing you to capture and visualize the lineage of your data pipelines. Alation provides a user-friendly interface and collaborative features for data discovery and understanding.
4. Collibra: Collibra is an enterprise data governance platform that includes metadata management and lineage features. It offers integration options with Snowflake and AWS services, enabling you to capture and track the lineage of your data assets. Collibra provides comprehensive data governance capabilities and collaboration features for data stewards and users.

These tools provide various features beyond lineage tracking, such as data discovery, data quality management, and data governance. Consider your specific requirements, integration needs, and budget while selecting the tool that best fits your use case.