Managing Amazon EC2 Capacity for Development Workloads

Introduction

In this article, we'll explore three key mechanisms to help customers optimize and manage Amazon EC2 capacity for non-production environments:

• Instance Scheduling using the AWS Instance Scheduler on AWS
• Auto Scaling Groups (ASG) for dynamic capacity management
• Spot Instances for cost-optimized compute in non-critical workloads

The options discussed in this article work best for development workloads that don't require 24/7 availability and can tolerate occasional interruptions, making it less suitable for continuous integration pipelines that need guaranteed availability.

Capacity Management Strategy options

Option 1: Implement Instance Scheduling using AWS Instance Scheduler

The AWS Instance Scheduler automatically starts and stops (not terminate) EC2 instances based on predefined schedules, perfect for development environments that don't need 24/7 availability. You can deploy the AWS Instance Scheduler using CloudFormation templates. Configure schedules based on your development team's working hours - for example, automatically starting instances at 8 AM and stopping them at 6 PM on weekdays. The scheduler can be customized with various parameters including timezone settings to match your team's location. For more information about how to implement AWS Instance Scheduler, please refer to the Solution overview.

Important: To ensure reliable startup when capacity is constrained, complement the Instance Scheduler with the AWS Flexible Instance Starter. This solution:

• Listens for StartInstances failure events
• Automatically modifies failed instances to alternative instance types
• Retries the start operation with flexible instance selection
• Restores original instance types when instances are stopped

The flexible instance starter works independently of the Instance Scheduler provides configurable parameters to adapt the solution to your specific needs.

Benefits

• Automated Resource Management: Instance scheduling eliminates the manual overhead of starting and stopping development environments. The AWS Instance Scheduler automates the starting and stopping of Amazon EC2 instances, ensuring that resources are only running when needed. This automation works across multiple AWS accounts, making it valuable for larger development organizations.

Considerations

• Implementation Complexity: Setting up automated scheduling requires initial configuration effort. The scheduler is installed with AWS CloudFormation templates and requires customization with parameters such as schedule, service type, and timezone settings.
• Capacity Availability: When capacity is not available in your preferred instance type, scheduled starts will fail. This can disrupt development workflows if teams expect consistent morning availability. Complementing Instance Scheduler with Flexible Instance Starter can help in mitigating the capacity issues. Your workloads must be suitable to run on various instance types to benefit from this flexibility.

Option 2: Use Auto Scaling Groups with Scheduled Scaling Actions for OnDemand and Spot instances

Auto Scaling Groups with scheduled scaling actions provide automated capacity management by launching and terminating instances based on predefined schedules. This approach is ideal for development workloads that follow predictable patterns - scaling up during business hours when developers are active and scaling down during nights and weekends to minimize costs.

To maximize capacity availability and cost efficiency, create Auto Scaling Groups that combine On-Demand and Spot Instances with Attribute-Based Instance type selection to express requirements as vCPU, memory, and storage attributes rather than specific instance types.

Configure your Auto Scaling Group to use the price-capacity-optimized allocation strategy for Spot Instances, which automatically provisions instances from the most-available Spot capacity pools with the lowest possible price. This approach automatically uses newer generation instances and provides broader capacity access.

Beyond instance flexibility, deploy your Auto Scaling Groups across multiple Availability Zones in a Region as each AZ has its own distinct instance capacity pools that you can leverage to scale your application fleets. Multi-AZ Auto Scaling groups also handle Insufficient Capacity Errors automatically. Combining AZ flexibility with instance flexibility will widen the capacity pools available to scale out your fleets of instances.

To further enhance availability, enable Capacity Rebalancing on the Auto Scaling Group to proactively replace Spot Instances that are at elevated risk of interruption, maintaining your desired capacity and application availability.

Benefits:

• Dynamic Scheduled Scaling Capabilities: Auto Scaling Groups provide automated scheduling for launching and terminating instances, allowing teams to automatically start environments during business hours and shut them down after hours to optimize costs.
• Cost Optimization: Combining instance scheduling with Spot Instances provides access to EC2's unused capacity and can save up to 90% off On-Demand pricing. When combined with automated scheduling, development teams can achieve substantial cost reductions by ensuring resources only run when needed.
• Proactive Interruption Handling: Capacity Rebalancing automatically responds when Spot Instances approach termination, launching replacement instances before interruptions occur. Capacity Rebalancing launches replacement instances when AWS emits rebalance recommendations sooner than the two-minute Spot Instance interruption notice. This proactive approach maintains capacity but temporarily exceeds desired count during transition.

Considerations:

• On-Demand capacity: Instances Capacity isn't guaranteed to always be available. There might be short windows of time when AWS doesn't have enough On-Demand capacity available to fulfill your specific request as the availability of On-Demand capacity changes frequently.
• Interruption Tolerance: Spot Instances may be reclaimed with notification when AWS needs the capacity back. This makes them suitable for asynchronous, stateless, and interruptible workloads typical in development environments but requires architectural considerations for state management. Your application should handle dynamic instance changes and potential Spot Instance interruptions.

Conclusion

The proposed capacity management solution addresses the core needs of development workloads by combining intelligent scheduling, dynamic scaling, and cost-optimized compute resources.

By implementing instance scheduling with Auto Scaling Groups using price-capacity-optimized Spot Instances, development teams can achieve significant cost savings while maintaining the flexibility and performance needed for effective software development and testing.

This approach is particularly effective for running periodic benchmarks or load tests, where the price-capacity-optimized allocation strategy with Auto Scaling Groups provides both cost efficiency and reduced interruption risk.

The automation capabilities ensure that these strategies can be implemented without adding operational overhead to development teams.

For further reading, explore the AWS Well-Architected Framework's cost optimization pillar, the EC2 Spot Instances best practices guide, and the comprehensive documentation on Auto Scaling Group allocation strategies to deepen your understanding of advanced capacity management techniques.

For more information:

• Create mixed instances group using attribute-based instance type selection
• Best practices for Amazon EC2 Spot
• EC2 instance rebalance recommendations

State Management Strategy - Warning:

For Spot-based development environments and scheduled AutoScaling, store all persistent data externally using Amazon S3 for application state and Amazon EFS for shared file systems. Configure instance User Data to automatically restore state on launch. Design applications to be stateless where possible, treating instances as ephemeral and replaceable. This approach ensures rapid recovery from both scheduled scaling operations and unexpected Spot interruptions.

Observability

Amazon EC2 Capacity Manager provides a unified dashboard to monitor and analyze Amazon EC2 usage across accounts and regions. It tracks On-Demand, Spot and Reserved capacity, helping identifying underused resources and cost optimization opportunities. You can also export detailed metrics for deeper analysis and capacity planning.

Implementing Amazon EC2 Capacity Manager in your capacity management for Development workloads can help in the following ways:

• Single central dashboard that enables you to have a cross-account and cross-Region capacity visibilty for On-Demand Instances, Spot Instances, and Capacity Reservations that help you analyzing usage patterns across instance types and teams, and providing insights into Spot Instance interruption patterns.
• Monitor and identify optimization opportunities for Spot Instance workloads by having the visibility into Spot instances and amount of time the Spot instances run before being interrupted by AWS to fine-tune mixed instance policies.
• Analyze collected metrics data by vCPUs, instance counts, or estimated costs to understand resource consumption patterns.
• Identify seasonal patterns (e.g., lower usage during holidays, higher during sprint planning)

This centralized visibility eliminates the need for development teams to manually check capacity across multiple services, enabling data-driven decisions about when and where to run development workloads most effectively.

For more information about Amazon EC2 Capacity Manager, you can refer to Monitor, analyze, and manage capacity usage from a single interface with Amazon EC2 Capacity Manager blog.

CLI Implementation example steps for Option 2

Prerequisites: The used AMI should have aws-cli installed, like AL2023 AMIs.

Note 1: Please replace the default values of vpc-id, region-code and account-id with your values.

• Create the S3 bucket to store application state data that needs to persist across instance replacements.

aws s3 mb s3://dev-workload-state-<account-id> --region <region-code>

• Create the EC2 Security group that we will use later to launch the instances. By default, AWS security groups allow all outbound traffic, which enables instances to communicate with AWS services like SSM (Systems Manager) for remote access and S3 for state backup and restoration.

aws ec2 create-security-group \
  --group-name dev-workload-sg \
  --description "Security group for dev workload instances" \
  --vpc-id <vpc-id> \
  --region <region-code>

• Create IAM instance profile: This step establishes the IAM role and instance profile that grants EC2 instances the necessary permissions to function properly. The role includes three key policy attachments: AmazonSSMManagedInstanceCore for Systems Manager access (enabling secure shell access without SSH keys), S3 permissions for reading and writing application state data, and Auto Scaling permissions for instances to describe their lifecycle state and complete lifecycle actions during termination.

# Create IAM instance profile with required permissions:
cat > instance-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "ec2.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

# Create the IAM role:
aws iam create-role \
  --role-name dev-workload-role \
  --assume-role-policy-document file://instance-trust-policy.json
  
# Attach SSM managed policy:
aws iam attach-role-policy \
  --role-name dev-workload-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
  
# S3 access policy to access the state-persistance S3 bucket:
cat > s3-state-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:PutObject",
      "s3:ListBucket"
    ],
    "Resource": [
      "arn:aws:s3:::dev-workload-state-<account-id>",
      "arn:aws:s3:::dev-workload-state-<account-id>/*"
    ]
  }]
}
EOF

aws iam put-role-policy \
  --role-name dev-workload-role \
  --policy-name S3StateAccess \
  --policy-document file://s3-state-policy.json
  
# AutoScaling lifecycle permissions
cat > asg-lifecycle-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "autoscaling:DescribeAutoScalingInstances",
      "autoscaling:CompleteLifecycleAction"
    ],
    "Resource": "*"
  }]
}
EOF

aws iam put-role-policy \
  --role-name dev-workload-role \
  --policy-name AutoScalingLifecyclePolicy \
  --policy-document file://asg-lifecycle-policy.json  
  
# Create the EC2 Instance profile
aws iam create-instance-profile \
  --instance-profile-name dev-workload-instance-profile  
  
# Attach the Role to the Instance Profile
aws iam add-role-to-instance-profile \
  --instance-profile-name dev-workload-instance-profile \
  --role-name dev-workload-role

• Create the EC2 user-data script with state restoration: This script first restores any existing application state from S3, then sets up three critical monitoring and backup mechanisms: a spot-monitor service that continuously checks for Spot rebalance recommendations and interruption notices every 5 seconds (refreshing authentication tokens every 5 hours), a systemd timer that performs periodic backups every 15 minutes, and a shutdown service that ensures state is backed up before the instance terminates.
• Note 1: I am saving my application state (like app logs) data under /app/state folder. Please consider to edit it based on your application.
• Note 2: In this example, I am using S3 to store the application state, but consider additionally using Amazon EFS to store the data.

cat > user-data.sh <<'EOF'
#!/bin/bash
S3_BUCKET="dev-workload-state-<account-id>"
ASG_NAME="dev-workload-asg"
REGION="<region-code>"

# Create directory and restore state
mkdir -p /app/state #this is an example of my application state folder
aws s3 sync s3://${S3_BUCKET}/app-state/ /app/state/

# Lifecycle hook handler
cat > /usr/local/bin/lifecycle-handler.sh <<'LIFECYCLEOF'
#!/bin/bash
S3_BUCKET="dev-workload-state-<account-id>"
ASG_NAME="dev-workload-asg"
REGION="<region-code>"
INSTANCE_ID=$(ec2-metadata --instance-id | cut -d " " -f 2)

# Backup state
aws s3 sync /app/state/ s3://${S3_BUCKET}/app-state/

# Only complete lifecycle action if one exists
aws autoscaling describe-auto-scaling-instances \
  --instance-ids ${INSTANCE_ID} \
  --region ${REGION} \
  --query 'AutoScalingInstances[0].LifecycleState' \
  --output text | grep -q "Terminating:Wait" && \
aws autoscaling complete-lifecycle-action \
  --lifecycle-hook-name backup-state-hook \
  --auto-scaling-group-name ${ASG_NAME} \
  --lifecycle-action-result CONTINUE \
  --instance-id ${INSTANCE_ID} \
  --region ${REGION}
LIFECYCLEOF

chmod +x /usr/local/bin/lifecycle-handler.sh

# Monitor for Spot events
cat > /usr/local/bin/spot-monitor.sh <<'SPOTEOF'
#!/bin/bash
BACKED_UP=false
TOKEN_REFRESH=0

while sleep 5; do
  # Refresh token every 5 hours
  if [ $TOKEN_REFRESH -eq 0 ]; then
    TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" 2>/dev/null)
    TOKEN_REFRESH=3600
  fi
  TOKEN_REFRESH=$((TOKEN_REFRESH - 1))
  
  REBALANCE=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/events/recommendations/rebalance 2>/dev/null)
  INTERRUPTION=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/spot/instance-action 2>/dev/null)
  
  if [ ! -z "$REBALANCE" ] || [ ! -z "$INTERRUPTION" ]; then
    if [ "$BACKED_UP" = false ]; then
      /usr/local/bin/lifecycle-handler.sh
      BACKED_UP=true
    fi
  fi
done
SPOTEOF

chmod +x /usr/local/bin/spot-monitor.sh

cat > /etc/systemd/system/spot-monitor.service <<'SVCEOF'
[Unit]
Description=Spot Event Monitor
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/spot-monitor.sh
Restart=always
[Install]
WantedBy=multi-user.target
SVCEOF

systemctl enable spot-monitor.service
systemctl start spot-monitor.service

# Periodic backup with systemd timer
cat > /etc/systemd/system/state-backup.service <<'BACKUPEOF'
[Unit]
Description=Backup state to S3
[Service]
Type=oneshot
ExecStart=/usr/bin/aws s3 sync /app/state/ s3://dev-workload-state-<account-id>/app-state/
BACKUPEOF
cat > /etc/systemd/system/state-backup.timer <<'TIMEREOF'
[Unit]
Description=Backup state every 15 minutes
[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target
TIMEREOF

systemctl enable state-backup.timer
systemctl start state-backup.timer

# Shutdown backup
cat > /etc/systemd/system/backup-shutdown.service <<'SHUTEOF'
[Unit]
Description=Backup on shutdown
DefaultDependencies=no
Before=shutdown.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/lifecycle-handler.sh
TimeoutStartSec=120
[Install]
WantedBy=shutdown.target
SHUTEOF
systemctl enable backup-shutdown.service
EOF

• Create the Launch Template: The template specifies instance requirements using attribute-based selection (2-4 vCPUs, 2-4GB memory) rather than specific instance types, providing flexibility to use various instance families and improving capacity availability.
• Note 1: Please replace ImageId with your AMI ID, and SecurityGroupIds with your security group created in step 2.
• Note 2: To convert the user-data.sh to base64 to be used in the EC2 user-data, you can use this command: USER_DATA_BASE64=$(base64 -i user-data.sh), then use the USER_DATA_BASE64 in the launch template as the below example.

aws ec2 create-launch-template \
  --region <region-code> \
  --launch-template-name dev-workload-template \
  --launch-template-data "{
    \"ImageId\": \"<ami-id>\",
    \"InstanceRequirements\": {
      \"VCpuCount\": {\"Min\": 2, \"Max\": 4},
      \"MemoryMiB\": {\"Min\": 2048, \"Max\": 4096},
      \"CpuManufacturers\": [\"intel\", \"amd\"]
    },
    \"IamInstanceProfile\": {
      \"Name\": \"dev-workload-instance-profile\"
    },
    \"SecurityGroupIds\": [\"<sg-id>\"],
    \"UserData\": \"${USER_DATA_BASE64}\",
    \"BlockDeviceMappings\": [{
      \"DeviceName\": \"/dev/xvda\",
      \"Ebs\": {
        \"VolumeSize\": 20,
        \"VolumeType\": \"gp3\",
        \"DeleteOnTermination\": true,
        \"Encrypted\": true
      }
    }],
    \"TagSpecifications\": [{
      \"ResourceType\": \"instance\",
      \"Tags\": [
        {\"Key\": \"Name\", \"Value\": \"dev-workload-instance\"},
        {\"Key\": \"Environment\", \"Value\": \"development\"}
      ]
    }]
  }"

• Create the AutoScaling group: The Auto Scaling Group is configured with a mixed instances policy that combines 50% On-Demand and 50% Spot Instances, using the price-capacity-optimized allocation strategy for Spot Instances. This strategy selects Spot capacity pools with both the lowest interruption risk and best pricing. The ASG is deployed across multiple Availability Zones to maximize capacity availability and handle Insufficient Capacity Errors automatically. Capacity Rebalancing is enabled, which proactively launches replacement instances when Spot Instances are at elevated risk of interruption, maintaining application availability. The group starts with zero capacity (min: 0, max: 10, desired: 0) to avoid costs until scheduled scaling actions activate it.

aws autoscaling create-auto-scaling-group \
  --region <region-code> \
  --auto-scaling-group-name dev-workload-asg \
  --mixed-instances-policy '{
    "LaunchTemplate": {
      "LaunchTemplateSpecification": {
        "LaunchTemplateName": "dev-workload-template",
        "Version": "$Default"
      }
    },
    "InstancesDistribution": {
      "OnDemandBaseCapacity": 0,
      "OnDemandPercentageAboveBaseCapacity": 50,
      "SpotAllocationStrategy": "price-capacity-optimized"
    }
  }' \
  --min-size 0 \
  --max-size 10 \
  --desired-capacity 0 \
  --vpc-zone-identifier "<subnet-id-1>,<subnet-id-2>,<subnet-id-3>" \
  --capacity-rebalance

• A lifecycle hook is configured to pause instance termination for up to 120 seconds, giving the instance time to complete state backup to S3 before shutdown. When an instance enters the terminating state (whether due to Spot interruption, capacity rebalancing, or scheduled scale-down), the lifecycle hook holds the instance in a "Terminating:Wait" state. The UserData script's lifecycle-handler detects this state, performs the S3 backup, and then completes the lifecycle action, signaling Auto Scaling that it's safe to proceed with termination. This ensures no application state is lost during instance replacements.

# Create lifecycle hook
aws autoscaling put-lifecycle-hook \
  --lifecycle-hook-name backup-state-hook \
  --auto-scaling-group-name dev-workload-asg \
  --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
  --default-result CONTINUE \
  --heartbeat-timeout 120 \
  --region <region-code>

• Create scheduled scaling actions: Two scheduled actions automate the scaling of development resources based on business hours. The scale-up action runs at 8 AM Monday through Friday (UTC timezone), setting desired capacity to 4 instances with a minimum of 2 and maximum of 10, ensuring development resources are available when teams start work. The scale-down action runs at 6 PM Monday through Friday, reducing desired capacity to 0 to eliminate costs during non-working hours and weekends. This automated scheduling can reduce development infrastructure costs by approximately 70% by ensuring instances only run during active development periods.

aws autoscaling put-scheduled-update-group-action \
  --region <region-code> \
  --auto-scaling-group-name dev-workload-asg \
  --scheduled-action-name scale-up-business-hours \
  --recurrence "0 8 * * MON-FRI" \
  --desired-capacity 4 \
  --min-size 2 \
  --max-size 6
  
aws autoscaling put-scheduled-update-group-action \
  --region <region-code> \
  --auto-scaling-group-name dev-workload-asg \
  --scheduled-action-name scale-down-after-hours \
  --recurrence "0 18 * * MON-FRI" \
  --desired-capacity 0 \
  --min-size 0 \
  --max-size 6

Finally, to verify that state persistence works correctly across instance replacements, you can test using several methods. First, scale up the Auto Scaling Group and create test data on a running instance by connecting via SSM Session Manager and creating files in the /app/state/ directory (e.g., echo "test data $(date)" | sudo tee /app/state/test.txt). Then verify the data is backed up to S3 using aws s3 ls s3://dev-workload-state-<account-id>/app-state/.

For testing Spot interruptions, you can use AWS Fault Injection Simulator (FIS) to simulate real Spot interruption scenarios, or use the open-source tool amazon-ec2-spot-interrupter which sends rebalance recommendations and interruption notices to your Spot instances. After the interruption, verify that a new instance launches automatically, restores the state from S3, and contains your test data.

For testing scheduled terminations or manual scale-down events, simply terminate an instance manually using aws ec2 terminate-instances --instance-ids <instance-id> or reduce the ASG desired capacity. The lifecycle hook will pause the termination, allowing the backup to complete before the instance shuts down. Once a replacement instance launches, connect to it and verify your test data was restored from S3. You can monitor the entire process by checking UserData execution logs at /var/log/cloud-init-output.log, spot-monitor service logs with sudo journalctl -u spot-monitor.service, and S3 backup timestamps to confirm backups occurred at the expected times (during interruption, every 15 minutes via timer, and at shutdown).