Letting AWS Auto Scaling (ASG) to Always Terminate the Oldest Instance by Implementing a Custom Termination Policy

Background: AWS Auto Scaling Groups (ASGs) offer various termination policies, including the OldestInstance policy, designed to retire the oldest instances first. However, ASGs prioritize an equal distribution of instances across Availability Zones (AZs). This behavior can sometimes lead to newer instances in one AZ being terminated before older instances in another AZ.

Needs: Customers often require a more predictable scaling and resource management approach that consistently terminates the oldest instance, regardless of AZ distribution.

Solution: To meet this need, a custom Lambda function was developed to implement a bespoke termination policy. This function leverages the EC2 DescribeInstances API to accurately identify the oldest instance in an ASG. Deployment is streamlined through a CloudFormation template, which automatically sets up the Lambda function and necessary IAM permissions, simplifying integration of this custom termination policy.

CloudFormation YAML File Content

The following YAML content outlines the CloudFormation template used to deploy this solution:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  LambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: LambdaExecutionPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: 'arn:aws:logs:*:*:*'
              - Effect: Allow
                Action:
                  - 'ec2:DescribeInstances'
                Resource: '*'
  MyLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import boto3
          import logging
          logger = logging.getLogger()
          logger.setLevel(logging.INFO)
          ec2 = boto3.client('ec2')
          def lambda_handler(event, context):
              logger.info("Received event: " + str(event))
              instance_ids = [instance['InstanceId'] for instance in event.get('Instances', [])]
              logger.info("Instance IDs: " + str(instance_ids))
              if instance_ids:
                  descriptions = ec2.describe_instances(InstanceIds=instance_ids)
                  instances = [i for r in descriptions['Reservations'] for i in r['Instances']]
                  oldest_instance = sorted(instances, key=lambda x: x['LaunchTime'])[0]['InstanceId']
                  logger.info("Oldest instance ID: " + oldest_instance)
                  return {'InstanceIDs': [oldest_instance]}
              else:
                  logger.info("No instances to process.")
                  return {'InstanceIDs': []}
      Runtime: python3.9
      Timeout: 30
  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !GetAtt MyLambdaFunction.Arn
      Principal: !Sub 'arn:aws:iam::${AWS::AccountId}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling'

Steps to Deploy the CloudFormation Template:

1. Save the Template:

   • Save the above YAML content to a file named terminate_oldest_template.yaml.

2. Access AWS Management Console:

   • Log into your AWS account and navigate to the AWS CloudFormation console.

3. Create a New Stack:

   • Select "Create stack" -> "With new resources (standard)".
   • Choose "Upload a template file", click "Choose file", and upload the terminate_oldest_template.yaml file.
   • Click "Next".

4. Specify Stack Details:

   • Enter a stack name, such as ASGTerminateOldestInstanceStack.
   • Click "Next".

5. Configure Stack Options:

   • Configure any necessary options, like tags or permissions. For most cases, default options are sufficient.
   • Click "Next".

6. Review and Launch:

   • Review all settings to ensure they are correct.
   • Acknowledge that AWS CloudFormation might create IAM resources, then click "Create stack".

7. Monitor Deployment:

   • Stay on the CloudFormation console to monitor the stack creation. Wait until the status changes to "CREATE_COMPLETE".

Change the termination policy for an Auto Scaling group:

• Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/, and choose Auto Scaling Groups from the navigation pane.

• Click the Auto Scaling group.

• On the Details tab, choose Advanced configurations, Edit.

• For Termination policies, choose Custom termination policy and then choose a Lambda function that created by CloudFormation. Check this AWS documentation for more details on custom termination policies.

References:

• AWS Predefined Termination Policies
• Configuring Custom Termination Policies