Keeping Unhealthy Auto Scaling (ASG) Instances (Detaching Unhealthy ASG Instance instead of Terminating)

Background: AWS Auto Scaling Groups (ASGs) monitor the health of instances and automatically replace unhealthy ones to ensure reliability and availability. This process involves terminating the unhealthy instances and launching new instance as replacements.

Customer Needs: Despite the efficiency of ASGs, there are scenarios where customers need to retain unhealthy instances for further analysis. For example, if an instance is marked as unhealthy by an Elastic Load Balancer (ELB), customer might want to investigate the root cause of the failure. Similarly, there might be a need to extract logs or other data from the instance to understand what went wrong.

Current Limitation: ASG's default behavior not allow to keep the unhealthy instances. While lifecycle hooks can delay termination, they only offer a temporary delay (up to 2 hours). Extending this period requires continuous API calls (RecordLifecycleActionHeartbeat), which is not an ideal or sustainable solution for long-term retention of unhealthy instances.

Solution: To address this challenge, I have developed a Lambda function that effectively bypasses the automatic replacement of unhealthy instances. The function works by suspending the "Replace Unhealthy" process in the ASG settings. It then monitors the health status of instances in the ASG every minute. Upon detecting an unhealthy instance, the Lambda function detaches it from the ASG, preventing its termination (check the unhealthy instance in EC2 console). And automatically launch a new replacement instance. Optionally, it can also notify customer through an SNS notification, providing details of the action taken and the instance affected. This solution not only preserves the unhealthy instances for investigation but also maintains the integrity and performance of the ASG by ensuring only healthy instances are active and serving traffic.

For the deployment, Lambda needs an IAM role and relevant permissions to work, it also needs EventBridge to trigger lambda every minute. The Lambda can also send SNS message. Configuring the whole solution manually is difficult and not customer-friendly. Further, I wrote a CloudFormation Template that can automatically configure it. Customer just need to provide the ASG name (support multiple ASGs) and SNS topic ARN (Optional).

Here is the CloudFormation Template:

AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation template to create a Lambda function with optional SNS topic and schedule it to run every minute using EventBridge to preserve unhealthy ASG instances.'

Parameters:
  SNSTopicARN:
    Type: String
    Default: ''
    Description: The ARN of the SNS topic for notifications (optional).

  ASGNames:
    Type: CommaDelimitedList
    Description: A comma-separated list of Auto Scaling Group names to monitor (Support multiple ASGs).

Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: LambdaExecutionPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  - autoscaling:DescribeAutoScalingGroups
                  - autoscaling:DetachInstances
                  - autoscaling:SuspendProcesses
                  - sns:Publish
                Resource: '*'

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import boto3
          from datetime import datetime
          import os

          autoscaling_client = boto3.client('autoscaling')
          sns_client = boto3.client('sns')

          ASG_NAMES = os.getenv('ASG_NAMES', '').split(',')
          SNS_TOPIC_ARN = os.getenv('SNS_TOPIC_ARN')
          MAX_DETACH = 20  # Maximum number of instances to detach at once (ASG API limitation)

          def process_asg(asg_name):
              response = autoscaling_client.describe_auto_scaling_groups(
                  AutoScalingGroupNames=[asg_name]
              )
              asg = response['AutoScalingGroups'][0]
              
              # Check and suspend ReplaceUnhealthy process if not already suspended
              if 'ReplaceUnhealthy' not in [process['ProcessName'] for process in asg['SuspendedProcesses']]:
                  autoscaling_client.suspend_processes(
                      AutoScalingGroupName=asg_name,
                      ScalingProcesses=['ReplaceUnhealthy']
                  )

              return [i['InstanceId'] for i in asg['Instances'] if i['HealthStatus'] != 'Healthy'], asg['DesiredCapacity']

          def lambda_handler(event, context):
              timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
              message_details = []

              for asg_name in ASG_NAMES:
                  unhealthy_instances, desired_capacity = process_asg(asg_name)
                  max_detach_by_percent = max(int(desired_capacity * 0.1),1) # Ensure minimum 1 if there are unhealthy instances
                  max_to_detach = min(max_detach_by_percent, MAX_DETACH, len(unhealthy_instances))
                  to_detach = unhealthy_instances[:max_to_detach]

                  if to_detach:
                      autoscaling_client.detach_instances(
                          InstanceIds=to_detach,
                          AutoScalingGroupName=asg_name,
                          ShouldDecrementDesiredCapacity=False
                      )
                      for instance_id in to_detach:
                          message_details.append(f"Detached unhealthy instance {instance_id} from ASG {asg_name} at {timestamp}")

              if message_details and SNS_TOPIC_ARN:
                  sns_message = "\n".join(message_details)
                  sns_client.publish(
                      TopicArn=SNS_TOPIC_ARN,
                      Message=sns_message,
                      Subject='Unhealthy ASG Instance Detached'
                  )
      Runtime: python3.8
      Timeout: 120
      Environment:
        Variables:
          SNS_TOPIC_ARN: !Ref SNSTopicARN
          ASG_NAMES: !Join [",", !Ref ASGNames]

  EventRule:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: 'rate(1 minute)'
      Targets:
        - Arn: !GetAtt LambdaFunction.Arn
          Id: "TargetFunction"
  
  LambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt LambdaFunction.Arn
      Action: 'lambda:InvokeFunction'
      Principal: 'events.amazonaws.com'
      SourceArn: !GetAtt EventRule.Arn

Outputs:
  LambdaFunctionArn:
    Description: "ARN of the Lambda function"
    Value: !GetAtt LambdaFunction.Arn

Steps to Deploy the CloudFormation Template:

1. Save the Template:

   • Save the above YAML content to a file named keep_unhealthy_ASG_instance.yaml.

2. Access AWS Management Console:

   • Log into your AWS account and navigate to the AWS CloudFormation console.

3. Create a New Stack:

   • Select "Create stack" -> "With new resources (standard)".
   • Choose "Upload a template file", click "Choose file", and upload the keep_unhealthy_ASG_instance.yaml file.
   • Click "Next".

4. Specify Stack Details:

   • Enter a stack name, such as KeepUnhealthyASGInstanceStack.
   • Specify the parameter, like ASGNames (A comma-separated list of Auto Scaling Group names to monitor) and The ARN of the SNS topic for notifications (optional). You can follow this doc to configure the SNS topic
   • Click "Next".
   • Screenshot for reference:

5. Configure Stack Options:

   • Configure any necessary options, like tags or permissions. For most cases, default options are sufficient.
   • Click "Next".

6. Review and Launch:

   • Review all settings to ensure they are correct.
   • Acknowledge that AWS CloudFormation might create IAM resources, then click "Create stack".

7. Monitor Deployment:

   • Stay on the CloudFormation console to monitor the stack creation. Wait until the status changes to "CREATE_COMPLETE".

Verify the result:

After the CloudFormation deploy complete, you can go to ASG console to verify the "Replace Unhealthy" process has suspended. You can also use below CLI to make one instance as unhealthy and then verify whether the unhealthy instance kept (in EC2 console) and new instance launch automatically in ASG:

aws autoscaling set-instance-health \
    --instance-id i-061c63c5eb45f0416 \
    --health-status Unhealthy

Script Limitations:

• Instance Detachment: The script is configured to detach a maximum of 10% of an ASG’s desired capacity or 20 instances, whichever is lower, at a single invocation. (You can modify the 10% vaule based on your use case.) This limitation is designed to mitigate the impact on service availability by preventing mass detachment of unhealthy instances.

• Health Status Diagnosis: The script does not provide specific reasons why an instance is marked as unhealthy. To diagnose the cause of an instance's health status, review the Elastic Load Balancer (ELB) and EC2 metrics. These metrics can offer insights into the reasons behind the instance's unhealthy state.

Disclaimer:

I have tested this script and it works well in my lab environment. However, please note this script is developed with the best effort, and while we strive for accuracy, we cannot guarantee its functionality. It is essential to thoroughly test the script in a test environment before deploying it in a production. If you have questions about this script, please leave a comment or contact AWS support team.