Elastic Beanstalk update-environment config changes fail with "Service:AmazonCloudFormation, Message:S3 error: Access Denied" (create + version deploys work)

Question

Setup

Elastic Beanstalk, platform 64bit Amazon Linux 2023 v4.13.3 running Docker
Region us-east-1, several environments sharing one Application Load Balancer
EB storage bucket elasticbeanstalk-us-east-1- — default encryption SSE-S3 (AES256), no KMS
Environments created/managed via Terraform (aws_elastic_beanstalk_environment)
Problem Any update-environment that changes configuration option settings (e.g. adding an aws:elasticbeanstalk:application:environment property) fails and rolls back:

Environment update is starting.
Updating environment 's configuration settings.
ERROR  Service:AmazonCloudFormation, Message:S3 error: Access Denied
ERROR  Failed to deploy configuration.
There are no CloudFormation UPDATE_FAILED stack-resource events — it fails at the EB service layer before CFN resource updates.

What works (the confusing part)

Creating an environment with the same option settings → works.
update-environment --version-label  (application-version deploys) → works.
Only configuration/option-setting updates fail.
Already ruled out

No SCPs (Organization has SCPs disabled) and no permission boundaries on the roles.
Calling principal has s3:* on the EB bucket (direct aws s3 calls succeed).
EB service role has an inline policy granting s3:GetObject/PutObject/ListBucket/ GetBucketLocation/GetBucketPolicy on the EB bucket — no change.
EB bucket policy grants s3:* to arn:aws:iam:::root AND an Allow to the cloudformation.amazonaws.com service principal — no change.
Bucket is SSE-S3 (no KMS Decrypt needed).
Objects under resources/environments/* are owned by the account's own canonical id (bucket ownership BucketOwnerPreferred).
No cf-templates-* bucket exists in the account/region.
No AccessDenied appears in CloudTrail management events during the failure (the denied S3 call seems to be a data-plane op not captured by default).
Question During an EB configuration update, which principal makes the S3 request, and what S3 action + bucket + key is being denied? (Creates and version-deploys don't hit it.) What is the correct permission/config so update-environment --option-settings succeeds?

Reproduction

aws elasticbeanstalk update-environment \
  --environment-name  \
  --option-settings Namespace=aws:elasticbeanstalk:application:environment,OptionName=DEPLOY_PROBE,Value=1 \
  --region us-east-1
# -> Service:AmazonCloudFormation, Message:S3 error: Access Denied ; rolls back
# The identical option-set applied at environment creation succeeds.

Riku_Kobayashi · Answer

Hello.

Why not try assigning "AWSElasticBeanstalkRoleCore" to the IAM role?  
This IAM policy is a managed policy used by Elastic Beanstalk to manage the service, so configuring it might make things work.  
https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSElasticBeanstalkRoleCore.html

Taka · Answer

When the template body (`TemplateBody`) exceeds a certain size (the limit for passing it directly inline, approximately 51,200 bytes), CloudFormation performs an internal process where it automatically creates an S3 bucket named `cf-templates--`, temporarily uploads the template there, and then executes the stack operation.
This aligns perfectly with your situation.

* Creating a new environment (create): The initial CFN template is relatively small and fits inline, so no S3 upload occurs → Success.
* Deploying only a version label: No CloudFormation stack update actually takes place (EB calls APIs equivalent to ASG/EC2/CodeDeploy directly, bypassing CFN) → Success.
* Changing configuration options: EB must reconstruct a larger CFN template reflecting the environment's "fully up-to-date state" and call `UpdateStack`. Since this template size exceeds the inline limit, CloudFormation attempts to create a new `cf-templates-*` bucket → If permissions for bucket creation (`s3:CreateBucket`) or subsequent actions like `PutObject` or `PutBucketPolicy` are missing, an "Access Denied" error occurs.

Furthermore, the fact you confirmed that no `cf-templates-*` bucket exists in the account/region strongly suggests that this automatic creation process has never succeeded. Had it succeeded even once in the past, the bucket would already exist.

Question 1:
This is a data plane operation (S3 bucket creation and object operations); as you noted, it is not recorded in default CloudTrail management events. To identify it definitively, you need to do one of the following:

Temporarily enable CloudTrail data events to log S3 `CreateBucket`, `PutObject`, and `PutBucketPolicy` events. By examining the event immediately following the configuration change, you can identify the specific principal (likely the service-linked role assumed by the CloudFormation service or the EB service role) and the action that were denied.

Question 2:

If this hypothesis is correct, you need to grant create and write permissions for buckets following the naming pattern `cf-templates-*` to the principal used internally by EB (typically the EB service role, or in some cases, the calling IAM role).

Elastic Beanstalk update-environment config changes fail with "Service:AmazonCloudFormation, Message:S3 error: Access Denied" (create + version deploys work)

-> Service:AmazonCloudFormation, Message:S3 error: Access Denied ; rolls back

The identical option-set applied at environment creation succeeds.

Add your answer

Relevant content