Glue data quality incremental run from console

Based on my research, you can schedule AWS Glue jobs from the console using cron expressions. This allows you to define a time-based schedule for your jobs with a minimum precision of 5 minutes. For instance, you could set up a job to run every day at a specific time, or at intervals throughout the day.

https://docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html

For incremental data processing, AWS Glue provides a feature called "job bookmarks". This feature allows AWS Glue to keep track of data that has already been processed, thereby preventing the reprocessing of old data during subsequent job runs. You can enable job bookmarks when setting up your AWS Glue job, and use them in combination with a "transformation context" parameter in your ETL code to manage state information for incremental data sets. This way, only the new data that has arrived since the last job run will be processed.

https://aws.amazon.com/blogs/big-data/process-data-with-varying-data-ingestion-frequencies-using-aws-glue-job-bookmarks/

Incremental Data quality validation with Glue Data Quality on Data Catalog is possible using partition push down predicates. As an example, if your data is partitioned by day and month, you can enter 'day=01 and day=02 and month=07' in partition predicate field that you have in your question. In this case, data quality is validated only for the first and second day of the month.

However, this field is static and does not take a parameter such as 'previous 7 days'. If you want to achieve this, instead of planning a single scheduled job that runs frequently, you need to run each job on-demand at the specific time after modification of partition predicate. So to achieve automation, you need to deploy a Cron job that triggers a Lamda that runs a Data Quality Rule with the right partition predicate at the right time and you need to keep track of the job runs.

Please note that you can run data quality jobs in Glue ETL jobs (this is called proactive data quality) as well as Glue Data Catalog Quality(the method you have mentioned). If you use this method, then you can benefit from job bookmarks. For the reference see : https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html

To filter a specific number of days' worth of data in AWS Glue, you can use PySpark's DataFrame API to filter the data. If your data includes a timestamp or date column, you can use the where function to filter the DataFrame based on this column. For example, to filter data for the previous day, you can use the following code:

from pyspark.sql import functions as F

df = Transform1.toDF()
df = df.where(F.col("time") == F.date_sub(F.current_date(), 1))
Transform2 = DynamicFrame.fromDF(df, glue_ctx=glueContext, name="df")

https://stackoverflow.com/questions/68505806/aws-glue-filter-date-field

In regards to your query about "Catalog partition predicate", AWS Glue allows you to use partition predicates to filter data on partitions directly on the partition metadata in the Data Catalog without having to list and read all the files in your dataset. This can save a significant amount of processing time if you only need a small subset of your data. The predicate expression can be any Boolean expression supported by Spark SQL. For instance, the predicate expression pushDownPredicate = "(year=='2017' and month=='04')" would load only the partitions in the Data Catalog that have both year equal to 2017 and month equal to 04​.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-partitions.html

There's also an option to use server-side partition pruning with the catalogPartitionPredicate option if you have a lot of partitions for a table and listing all the partitions from the catalog can incur additional time overhead. This option uses partition indexes in the AWS Glue Data Catalog and can make partition filtering much faster when you have millions of partitions in one table. You can use both push_down_predicate and catalogPartitionPredicate in additional_options together if your catalogPartitionPredicate requires predicate syntax that is not yet supported with the catalog partition indexes​.