Letting AWS Auto Scaling (ASG) to Always Terminate the Oldest Instance by Implementing a Custom Termination Policy

4 minute read
Content level: Intermediate
4

This article use AWS Auto Scaling Group (ASG) custom termination policies to consistently terminate the oldest instance, addressing limitations in default ASG settings where instance distribution across Availability Zones can override the oldest termination logic. It presents a solution integrating a Lambda function with CloudFormation for streamlined deployment and operational efficiency, ensuring predictable and manageable scaling activities.

Background: AWS Auto Scaling Groups (ASGs) offer various termination policies, including the OldestInstance policy, designed to retire the oldest instances first. However, ASGs prioritize an equal distribution of instances across Availability Zones (AZs). This behavior can sometimes lead to newer instances in one AZ being terminated before older instances in another AZ.

Needs: Customers often require a more predictable scaling and resource management approach that consistently terminates the oldest instance, regardless of AZ distribution.

Solution: To meet this need, a custom Lambda function was developed to implement a bespoke termination policy. This function leverages the EC2 DescribeInstances API to accurately identify the oldest instance in an ASG. Deployment is streamlined through a CloudFormation template, which automatically sets up the Lambda function and necessary IAM permissions, simplifying integration of this custom termination policy.

CloudFormation YAML File Content

The following YAML content outlines the CloudFormation template used to deploy this solution:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  LambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: LambdaExecutionPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: 'arn:aws:logs:*:*:*'
              - Effect: Allow
                Action:
                  - 'ec2:DescribeInstances'
                Resource: '*'
  MyLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import boto3
          import logging
          logger = logging.getLogger()
          logger.setLevel(logging.INFO)
          ec2 = boto3.client('ec2')
          def lambda_handler(event, context):
              logger.info("Received event: " + str(event))
              instance_ids = [instance['InstanceId'] for instance in event.get('Instances', [])]
              logger.info("Instance IDs: " + str(instance_ids))
              if instance_ids:
                  descriptions = ec2.describe_instances(InstanceIds=instance_ids)
                  instances = [i for r in descriptions['Reservations'] for i in r['Instances']]
                  oldest_instance = sorted(instances, key=lambda x: x['LaunchTime'])[0]['InstanceId']
                  logger.info("Oldest instance ID: " + oldest_instance)
                  return {'InstanceIDs': [oldest_instance]}
              else:
                  logger.info("No instances to process.")
                  return {'InstanceIDs': []}
      Runtime: python3.9
      Timeout: 30
  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !GetAtt MyLambdaFunction.Arn
      Principal: !Sub 'arn:aws:iam::${AWS::AccountId}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling'

Steps to Deploy the CloudFormation Template:

  1. Save the Template:

    • Save the above YAML content to a file named terminate_oldest_template.yaml.
  2. Access AWS Management Console:

    • Log into your AWS account and navigate to the AWS CloudFormation console.
  3. Create a New Stack:

    • Select "Create stack" -> "With new resources (standard)".
    • Choose "Upload a template file", click "Choose file", and upload the terminate_oldest_template.yaml file.
    • Click "Next".
  4. Specify Stack Details:

    • Enter a stack name, such as ASGTerminateOldestInstanceStack.
    • Click "Next".
  5. Configure Stack Options:

    • Configure any necessary options, like tags or permissions. For most cases, default options are sufficient.
    • Click "Next".
  6. Review and Launch:

    • Review all settings to ensure they are correct.
    • Acknowledge that AWS CloudFormation might create IAM resources, then click "Create stack".
  7. Monitor Deployment:

    • Stay on the CloudFormation console to monitor the stack creation. Wait until the status changes to "CREATE_COMPLETE".

Change the termination policy for an Auto Scaling group:

References:

profile pictureAWS
SUPPORT ENGINEER
Tim
published a month ago1103 views