Error when deploy EKS cluster from cdk pipeline

Question

I'm trying to deploy an eks cluster stack as part of my pipeline, its a self mutating pipeline, which includes stages to pull the code from gitlab, then build the cdk, update the stack, and then deploy the eks cluster

I can deploy the stack from my pc, but if i let codepipeline run the deployment using CloudFormationCreateUpdateStackAction it fails with the following error

awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454** - create failed - S3 error: Access Denied For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

a bit further on it fails with

The following resource(s) failed to create: [CustomAWSCDKOpenIdConnectProviderCustomResourceProviderRole517FED65, clusterAdmin9FD4BDDE, layer26FEA11C0D, nodeRoleA6A8ECDC, eksclusterControlPlaneSecurityGroupBAB2FC0A, eksclusterKubectlHandlerRole12CC1C9C, AWSCDKCfnUtilsProviderCustomResourceProviderRoleFE0EE867, eksclusterRoleAD55CC89, awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454]. Rollback requested by user.

when i look at the cloudtrail its complaining its unable to find an asset

```var pipeline = new Pipeline(this, "olapipeline", new PipelineProps
            {
                PipelineName = pipelineName,
                CrossAccountKeys = true,
                ArtifactBucket = artifactBucket,
                RestartExecutionOnUpdate = true,
            });
```

I have a number of stages before my deployment stage, first pulls the code using a custom action because we use gitlab, the next does a cdk synth on project, I'm not entirely sure how much code is needed, but i'll give it a go :)
           
##

##
```            pipeline.AddStage(new StageOptions
            {
                StageName = "DeployEksClusters",
                Actions = new IAction[]
                {
                    new CloudFormationCreateUpdateStackAction(new CloudFormationCreateUpdateStackActionProps
                    {
                        Account = props.LocalCluster.Account,
                        Region = props.LocalCluster.Region,
                        ActionName = "DeployLocalCluster",
                        StackName = props.LocalCluster.StackName,
                        TemplatePath = pipelineBuildOutput.AtPath("LocalCluster.template.json"),
                        DeploymentRole = cfRole,
                        CfnCapabilities = new CfnCapabilities[] { CfnCapabilities.NAMED_IAM, CfnCapabilities.AUTO_EXPAND },
                        Role = cfRole,
                        AdminPermissions = true
                    }),
              }
            });
```
the EKS stack looks like this

```            var cluster = new Cluster(this, "eks-cluster", new ClusterProps
            {
                Tags = tags,
                Mas ters Role = clusterAdmin,  
                Version = KubernetesVersion.Of("1.26"),
                Vpc = vpc,
                ClusterName = Props.ClusterName,
                VpcSubnets = new[] { subnets },
                AlbController = new AlbControllerOptions { Version = AlbControllerVersion.V2_5_1 },
                EndpointAccess = EndpointAccess.PRIVATE,
                DefaultCapacity = 0,
                KubectlLayer = new KubectlV26Layer(this, "layer26"),
            });

var managedNodeGroup = cluster.AddNodegroupCapacity("initial-node-group", new NodegroupOptions
            {
                MinSize = 1,
                MaxSize = 5,
                AmiType = NodegroupAmiType.AL2_X86_64,
                InstanceTypes = new[] { new InstanceType("m5.large") },
                CapacityType = CapacityType.SPOT,
                DiskSize = 40,
                NodeRole = nodeRole,
                Tags = tags,
            });
```

Thanks  in advance

Andrew

Answer

From your error, it looks like the role that is deploying the stack may not have permissions to the S3 bucket or to create the bucket.

Make sure you have provided either the proper permissions to allow your stack to create the bucket or to access the bucket.
Here's some additional information - https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-walkthroughs-managing-access-example1.html

Hope this helps.

Answer

Hello there,

Looking at the below error, the issue was in downloading of the nested stack's template while creating the nested stack.

```
awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454** - create failed - S3 error: Access Denied For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html 
```

`awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454` --> This looks like nested stack resource.

Possible causes of the issue and resolutions :

1. The CloudFormation service role of the stack does not have enough permission to pull the template of the nested stack from its S3 location.

Adding the below permission to CloudFormation service role may help here :

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}
```

You can optionally specify only the S3 bucket ARN where the nested stack's template is present in the "Resources" section of the above policy.

2. Bucket policy on the S3 bucket of the nested stack's template may be restricting the access from the AWS CloudFormation service role. Please check this and allow the CloudFormation service role to get the objects from this bucket.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html    --> [1]

3. If encryption is enabled for the S3 bucket and its using Customer managed KMS key, please provide the permission for the CloudFormation stack service role to use then KMS key.

Policy example is given below :

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey",
                "kms:GenerateDataKey*",
                "kms:Encrypt",
                "kms:ReEncrypt*",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}
```

If the KMS key is in different account, you may need to provide permission to access it in AWS KMS key policy for the Cloudformation stack role as well. Please refer to the documentation [2] for more information about KMS key policy.

I hope this information serves you. Further, troubleshooting about this requires checking IAM role policies, S3 bucket policy, KMS and other resources in AWS account. So, if you need additional help on this, you can feel free to reach out to AWS Premium Support.

--------------------------------------------------------------------------------
References :

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html

[2] https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html

Error when deploy EKS cluster from cdk pipeline

Relevant content