Glue Crawler on new data and no schema change

For your first question, you don't need to re-run the crawler every time your file is replaced with new data if the schema remains the same. Athena reads directly from the S3 location specified in the table definition, so as long as the new data follows the same schema and is placed in the same location with the same filename, Athena will automatically pick up the latest data without requiring a crawler run.

The crawler's primary purpose is to detect and update schema changes in the Data Catalog. Since your schema remains constant, running the crawler daily is unnecessary and may even create new table versions in the Data Catalog despite no actual schema changes.

For your second question regarding crawler performance metrics, there are several factors that could explain why your crawler occasionally takes longer to complete:

1. File size: If the replacement file is significantly larger on some days, the crawler will need more time to process it.

2. File complexity: Even with the same schema, more complex data within the file (more nested structures, more unique values) might require additional processing time.

3. AWS service load: The performance of AWS Glue services can vary based on overall service load.

4. Data format: The format of your data (JSON, CSV, Parquet, etc.) affects processing time, especially if the format requires more complex parsing.

To better understand crawler performance, you could monitor CloudWatch metrics for your Glue crawler, which include metrics like crawler runtime, and check CloudWatch Logs for any warnings or errors that might explain longer processing times.
Sources
AWS Glue crawler creating new version glue data catalog tables when no change in schema | AWS re:Post
Customizing crawler behavior - AWS Glue

No, you don't need to rerun the Glue Crawler each time for new data—Athena can query newly added S3 files automatically if the schema doesn't change, since it uses partition projection or the table's metadata location pointing to the S3 prefix. However, the Crawler is only needed when the schema/structure changes; for daily file replacements with identical schema, disable the Crawler and rely on Athena's native S3 scanning. To monitor Crawler performance, check CloudWatch metrics like CrawlerRuntimeInSeconds, DataCatalogUpdates, and TablesUpdated—the ~2-minute runs likely occur when the Crawler performs a full scan (even with unchanged schema) or encounters S3 list-object throttling; you can reduce runtime by limiting the Crawler to specific S3 prefixes and scheduling it only when schema changes are expected.