Maximum number of workers vs glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors

Question

I have mentioned 4 as the Maximum number of workers when defining the Glue Spark Job of G1X worker type . 
If I check the Cloudwatch Job monitors , I see some these below metrics touching 20 + in the line graph .

1. glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors
2. glue.driver.ExecutorAllocationManager.executors.numberAllExecutors

My understanding is, for G1X worker type , there is only 1 executor per worker . Can someone explain if I am reading the chart right or if there is something missing ?

This is the line graph for 3 jobs , each having just 4 workers configured.
![Enter image description here](/media/postImages/original/IM7pdSUHl0ReCC_TJgCuJmtQ)

Chaitu · Accepted Answer

Please note that the metric- **glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors** does not depend upon the number of workers configured by you for the job. It is dynamically calculated as the job progresses. Generally this metric has a higher value at the beginning of the job. It just shows the maximum number of executors that are required in current moment to satisfy the workload. This is a spark related metric and it is calculated by making use of number of partitions, number of tasks per executor.

Please refer this [documentation](https://docs.aws.amazon.com/glue/latest/dg/monitoring-awsglue-with-cloudwatch-metrics.html) for more details regarding various metrics of a Glue job.

You can have a look at this [documentation](https://docs.aws.amazon.com/glue/latest/dg/monitor-debug-capacity.html) which shows how various calculations involving executors.

Maximum number of workers vs glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors

Conteúdo relevante