How do I retrieve the total worker count that's configured for auto scaling in Amazon MWAA?

2 minute read
1

I want to retrieve the total worker count that's configured for auto scaling in Amazon Managed Workflows for Apache Airflow (Amazon MWAA).

Short description

The number of initial workers is equal to the minimum number of workers that you set. The workers beyond the initial minimum are the additional workers that Amazon MWAA automatically adds to manage increased task demands.

To retrieve your total number of additional worker instances in Amazon MWAA, check the cluster metrics for AdditionalWorker on the Amazon CloudWatch console. Amazon MWAA uses Amazon Elastic Container Service (Amazon ECS) on AWS Fargate containers to scale the worker fleet on demand.

Resolution

To retrieve the total worker count in Amazon MWAA, complete the following steps:

  1. Open the CloudWatch console, and then choose your Amazon MWAA namespace.
  2. On the All metrics tab, choose Cluster dimension.
  3. Under Cluster dimension, for AdditionalWorker, choose either the CPUUtilization or MemoryUtilization metric.
    Note: Amazon MWAA provides CPU and memory utilization metrics for Amazon ECS containers and saves the metrics for 15 months.
  4. From the Graphed metrics tab, set Period to 1 minute, and set Statistic to Sample Count.
AWS OFFICIAL
AWS OFFICIALUpdated 3 months ago
2 Comments

What is the definition of "Additional Worker"? If I run 4 workers minimum and 12 maximum, is "Additional Worker" = 4 + "Additional Worker" to figure out the original question of total worker count?

replied 4 months ago

If you want to know the total worker, make the same graph but on Base Worker, and create a new expression to add up base worker count and additional worker count.

replied 2 months ago