How do I troubleshoot issues related to Secrets Manager secrets in Amazon ECS?

Short description

If you don't place a task with a Secrets Manager secret in Amazon ECS, then you might receive one of the following error messages:

"ResourceInitializationError error on AWS Fargate AccessDenied error on Amazon Elastic Compute Cloud (Amazon EC2)"

"The task is unable to retrieve secrets from Secrets Manager and fails."

"ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 1 time(s): failed to fetch secret"

"AccessDeniedException: User: arn:aws:sts::XXXXXXXX:assumed-role/test-ecs-execution-role-no-secrets/XXXXXXXXXXX is not authorized to perform:"

These error messages occur because of one of the following reasons:

• The Amazon ECS task execution role doesn't have the required permissions to access Secrets Manager secrets.
• Secrets Manager can't find the specified secret.
• The name of the secret ends with a hyphen followed by six characters and you use a partial Amazon Resource Name (ARN) to search for a secret.
• There are issues with your Amazon Virtual Private Cloud (Amazon VPC) networking configuration.

To troubleshoot the errors for Amazon ECS tasks that fail to start, use the AWSSupport-TroubleshootECSTaskFailedToStart runbook. Then, refer to the relevant troubleshooting steps for your issue.

Resolution

Prerequisite: Make sure that your AWS Identity and Access Management (IAM) user or role has the required permissions to use the runbook. For more information, see the Required IAM permissions section of AWSSupport-TroubleshootECSTaskFailedToStart.

Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook in the same AWS Region where your Amazon ECS cluster resources are located.

Use the most recently failed Task ID. If the failed task is part of an Amazon ECS service, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during the automation execution. By default, stopped Amazon ECS tasks are visible for 1 hour after entering the Stopped state.

Run the AWS Systems Manager runbook

Complete the following steps:

1. Open the Systems Manager console.
2. In the navigation pane, choose Documents.
3. In the search bar, enter TroubleshootECSTaskFailedToStart.
4. Select the AWSSupport-TroubleshootECSTaskFailedToStart document.
5. Choose Execute automation.
6. Choose Execute.
7. After the automation completes, review the detailed results in the Outputs section and then, use one of the following manual troubleshooting steps.

Check the Amazon ECS task execution role permissions

Complete the following steps:

1. Open the IAM console.
2. In the navigation pane, choose Roles.
3. Search the list of roles for the task execution role that your Amazon ECS tasks use.
4. Confirm that you granted the task execution role the required permissions to access the Secrets Manager resource.

Check that the Secrets Manager secret exists

Complete the following steps:

1. Open the Secrets Manager console.
2. In the navigation pane, choose Secrets, and then choose the secret that your Amazon ECS tasks use.
3. Confirm that your Amazon ECS task definition includes these specific secret names. If you specified additional parameters in the secret, then confirm that your task definition includes the json-key/version-stage/version-id that's specified in the secret.

Check the name and ARN of the Secrets Manager secret

Check if your secret name ends with a hyphen followed by six characters. For example, myappsecret-xxxxxx. Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN. It's a best practice not to end your secret name with a hyphen followed by six characters.

To resolve this issue, use the full ARNs, including the additional hyphen and six characters, of the secrets from Secrets Manager.

Check your Amazon VPC networking configuration

Create the interface Amazon VPC endpoints for Secrets Manager when the following conditions are true:

• You use a task definition that references the Secrets Manager secrets to retrieve sensitive data for your containers.
• You use interface Amazon VPC endpoints.

Also, be sure that the security groups for your Amazon VPC endpoints allow the Amazon ECS infrastructure to use the endpoints.

To check if the Amazon VPC endpoint for Secrets Manager exists, complete the following steps:

1. Open the Amazon VPC console.
2. In the navigation pane, choose Endpoints.
3. Make sure that you specify the Amazon VPC endpoint com.amazonaws.example-region.secretsmanager for Secrets Manager.
4. Select the endpoint, and then choose the Subnets tab. Make sure that the subnets for this endpoint include the endpoints your Amazon ECS task uses.
   Note: If there are no subnets, then choose Manage Subnets. Select the subnet based on its Availability Zone, and then choose Modify Subnets.

To confirm that the security group attached to the endpoint com.amazonaws.example-region.secretsmanager allows incoming connections on port 443 from the Amazon ECS tasks, complete the following steps:

1. Open the Amazon VPC console.
2. In the navigation pane, choose Endpoints.
3. Select the endpoint.
4. Choose the Security Groups tab.
5. Choose the Group ID for the security group that you want to check.
6. Choose the Inbound rules tab.
7. Verify that the list of inbound rules includes a rule that allows 443 connections from your Amazon ECS tasks.

Related information

How can I securely pass secrets or sensitive information to containers in an Amazon ECS task?

Troubleshooting AWS Secrets Manager