By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How do I generate a log bundle for my Amazon EKS instances?

2 minute read
1

I'm troubleshooting an Amazon Elastic Kubernetes Service (Amazon EKS) instance. I need to collect all the relevant Amazon EKS logs that are associated with the instance.

Short description

Use the AWSSupport-CollectEKSInstanceLogs runbook to collect your Amazon EKS logs.

Important: For the automation to work, you must install and run AWS Systems Manager Agent (SSM Agent) on your Amazon EKS instance.

Resolution

Note: The following resolution works only for worker nodes in Amazon EKS for Linux.

To generate a log bundle for Amazon EKS instances, complete the following steps:

  1. Open the AWS Systems Manager console.
  2. In the navigation pane, choose Automation.
  3. Choose Execute automation.
  4. On the Owned by Amazon tab, in the Automation document search box, enter EKSInstanceLogs. Then, select AWSSupport-CollectEKSInstanceLogs.
  5. Choose Next.
  6. On the Automation Inputs page, for EKSInstanceId, enter your target Amazon EKS instance ID.
  7. Set the LogDestination and AutomationAssumeRole parameters.
    Note: The log bundle is uploaded to the Amazon Simple Storage Service (Amazon S3) bucket that's specified in the LogDestination field. If you didn't specify a bucket, then you can retrieve the log bundle from the instance. The log bundle is saved locally in the /var/log/ path.
  8. Choose Execute automation.

When you run AWSSupport-CollectEKSInstanceLogs, use the following best practices and requirements:

  • It's a best practice to attach the Amazon managed AmazonSSMManagedInstanceCore policy to the relevant AWS Identity and Access Management (IAM) role. The IAM role is for the Amazon Elastic Compute Cloud (Amazon EC2) instance that's targeted for automation.
  • If you provide the S3 bucket name in the LogDestination field before the automation runs, then the instance profile can write to the bucket.
  • To run the automation and send the command to the instance, you must have at least the ssm:ExecuteAutomation and ssm:SendCommand permissions.
  • To read the automation output, you must have the ssm:GetAutomationExecution permission.
  • For Amazon Linux 2, IMDS endpoints support both IMDSv2 and IMDSv1 by default. If you choose to configure IMDSv2, then the other version no longer works. Because AWSSupport-CollectEKSInstanceLogs uses IMDSv1, you might notice a Pending step when the collected logs are uploaded.

Related information

Configure instance permissions for Systems Manager

EKS Logs Collector on the GitHub website

Topics
ContainersEnd User Computing
Tags
Amazon Elastic Kubernetes ServiceAWS Support Automation Workflows
Language
English

Related videos

Watch Aman's video to learn more (2:41)
Watch Aman's video to learn more (2:41)
AWS OFFICIAL
AWS OFFICIALUpdated a year ago
3 Comments

Please note, when uploading EKS logs by executing the SSM document AWSSupport-CollectEKSInstanceLogs. Currently the SSM document requires the EC2 instance to use Metadata v1, when the recommended security best practice is to use Metadata v2 whenever possible.

You can confirm that the instance requires Metadata v2 by running:

aws ec2 describe-instances --instance-ids <Instance ID> | grep HttpTokens

"HttpTokens": "required", When IMDSv2 is required, you must send a session token with any instance metadata retrieval requests. In this state, retrieving the IAM role credentials always returns IMDSv2 credentials; IMDSv1 credentials are not available.

You need to follow a workaround in order to run the SSM document targeting instances which requires Metadata v2 ,

Select the automation AWSSupport-CollectEKSInstanceLogs

Click in Actions -> Clone Document

Replace the line:

From:

"region=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/\\(.*\\)[a-z]/\\1/')",

To:

"imds_token=$(curl -X PUT \"http://169.254.169.254/latest/api/token\" -H \"X-aws-ec2-metadata-token-ttl-seconds: 360\")",

"region=$(curl -H \"X-aws-ec2-metadata-token: $imds_token\" -f -s --max-time 10 --retry 5 http://169.254.169.254/latest/meta-data/placement/region)",

Click in Create Automation Execute the new Automation Copy-AWSSupport-CollectEKSInstanceLogs

profile pictureAWS
maicon-alegre
replied a year ago

I can confirm there is already a issue opened to EKS Service Team to update the SSM document AWSSupport-CollectEKSInstanceLogs to use IMDSv2 and I reinforced other customers are requesting this to be fixed.

profile pictureAWS
maicon-alegre
replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
EXPERT
AWS Official
replied a year ago

Relevant content