AgentCore CreateGateway internal GetPolicyEngine validation denied despite IAM simulator Allow and documented KMS permissions

So AWS's own documentation establishes that Policy in AgentCore error messages can name one resource or permission while the actual gap is a different one. Your error naming GetPolicyEngine is therefore not proof that GetPolicyEngine is what was denied.

The part that will still block you after you add them

AuthorizeAction and PartiallyAuthorizeActions are documented as requiring both ARNs in the resource array:

Both AuthorizeAction and PartiallyAuthorizeActions require access to BOTH the policy-engine and gateway resources.

This is a problem for your exact-ARN scoping discipline during CreateGateway, because the Gateway ARN does not exist yet at the moment the validation runs. You cannot scope to an ARN for a resource that is being created. You have two options: use gateway/* (the docs explicitly offer the wildcard), or pre-compute the deterministic Gateway ARN if your naming makes that possible. Given you're deploying via CloudFormation and the Gateway ID is service-generated, gateway/* on those two actions is the realistic choice. Note that the troubleshooting section's own remediation snippet uses policy-engine/* and gateway/* together.

This also explains something in your report that would otherwise be puzzling: your control test succeeded from the CloudFormation execution role, which you describe as having ARN-prefix-scoped permissions rather than exact-ARN scoping. Prefix scoping tolerates a not-yet-existent Gateway ARN; exact scoping does not.

Why your IAM simulation didn't catch this

You simulated bedrock-agentcore:GetPolicyEngine and got allowed with AllowedByPermissionsBoundary: true. That result is correct and also irrelevant to the two actions you never simulated. Before changing anything, run the simulator against AuthorizeAction and PartiallyAuthorizeActions with the Gateway ARN as the resource — I'd expect implicitDeny, which would confirm this in one command.

Worth knowing more generally: simulate-principal-policy evaluates identity policies, boundaries, and SCPs. It does not evaluate KMS grants, and it cannot reproduce context values that a service injects at call time. For a grant-mediated KMS failure it will return allowed while the real call fails, so it can't be used to clear the KMS side of this.

Finding 2: Policy in AgentCore decrypts through KMS grants created via FAS, not through your role's identity policy

This is the more consequential finding, and it invalidates the model your KMS configuration is built on. From Encrypt your AgentCore policy engine with a customer-managed KMS key:

Policy in AgentCore uses a grant-based model to access your customer managed key. When you create a policy engine with a customer managed key, Policy in AgentCore [...] creates two AWS KMS grants on your behalf. To create these grants, Policy in AgentCore calls kms:CreateGrant using your identity through a Forward Access Session (FAS). Because the CreateGrant call is made with your credentials via FAS, your AWS KMS key policy must grant kms:CreateGrant permission to your AWS account principal (not to a service principal). The grants themselves then allow Policy in AgentCore to perform cryptographic operations on your behalf without requiring further caller involvement.

Two grants are created, with distinct roles:

Grant	Purpose	Operations
Policy management grant	Creating, reading, updating, deleting policies	encrypt, decrypt, generate data key
Policy evaluation grant	Runtime Cedar evaluation	decrypt, re-encrypt

Both are constrained by the encryption context key aws:bedrock-agentcore-policy:policy-engine-arn.

The practical consequence: the cryptographic operations the service performs are authorized by the grants, not by a kms:Decrypt statement on your Gateway execution role. Your Gateway-role KMS statement, with its kms:ViaService and exact-encryption-context conditions, is operating at a layer that is not the one gating the service's decryption. That's consistent with your own observation that the DryRun events return DryRunOperationException with the expected encryption context — the operation would succeed on the service's authorization path.

The docs do state that the calling identity needs key access as well:

to create an encrypted policy engine with a customer managed key, or to make API calls to a policy engine encrypted by a customer managed key, the IAM user or role which makes the call must also have access to the key.

So a kms:Decrypt grant on the caller isn't wrong to have. But it is not the authorization path for the service-side decryption, and tightening it with two simultaneous StringEquals conditions gives you a very narrow statement whose failure mode is indistinguishable from the IAM gap in Finding 1.

The check I'd run first on the KMS side

Your Policy Engines are ACTIVE, which implies grant creation already succeeded. Confirm it directly:

aws kms list-grants \
  --key-id arn:aws:kms:ap-south-1:<ACCOUNT_ID>:key/<KMS_KEY_ID> \
  --region ap-south-1 \
  --query 'Grants[].{Name:Name,GranteePrincipal:GranteePrincipal,Ops:Operations,EncCtx:Constraints}' \
  --output json

You should see two grants per encrypted policy engine, with encryption-context constraints naming the policy engine ARN. If they're present, grant creation and the account-principal kms:CreateGrant delegation are working, and the KMS layer is likely not your blocker. If they're missing or there's only one, that's a real finding — and note the docs' warning that revoking only one of the two leaves the service in an inconsistent state.

One thing to verify on your permissions boundary while you're there: you describe it as allowing kms:Decrypt on tagged keys, with no mention of kms:CreateGrant. Because CreateGrant is called through FAS with the creating principal's credentials, that principal's boundary must permit kms:CreateGrant. It evidently did for your existing engines, but it will matter for any future engine creation, and it's a common omission.

Direct answers to your questions

Q1 — What kms:ViaService value does AgentCore provide? The documented format is bedrock-agentcore.REGION.amazonaws.com, where REGION is where the policy engine lives. For you that is bedrock-agentcore.ap-south-1.amazonaws.com. The docs give this for the key policy; the internal value used during the GenesisPolicyEngineCheck session is not separately documented, and as you found, CloudTrail doesn't expose it. Only Support can confirm the effective value on that specific internal path.

Q2 — Which authorization path does the internal validation use? For the KMS side the documented mechanism is KMS grants created through a Forward Access Session, and the docs are explicit that kms:ViaService "applies for Forward Access Sessions (FAS)." Grants are the service's path to crypto operations; FAS is how the grants get created using your identity. The docs do not describe the authorization path of the GetPolicyEngine call made under GenesisPolicyEngineCheck, and I won't guess at it — but your CloudTrail evidence that the call is made under the assumed-role session with no session policy is consistent with the assumed Gateway-role identity being evaluated directly, which is precisely why the role needs the full three-permission set.

Q3 — Which principals must authorize the decryption? For kms:CreateGrant, your AWS account principal in the key policy, explicitly not a service principal. After that, the two grants authorize the service's operations, so no additional per-call principal authorization is involved. Separately, the identity making API calls against an encrypted policy engine needs key access. Your key policy delegating to the account root is the right shape for the first part.

Q4 — Why is the role denied when its policy and boundary permit the actions? Because, on the evidence, the actions its policy permits are not the complete set the service requires. AuthorizeAction and PartiallyAuthorizeActions are missing, and the documented behaviour of that omission is that the Gateway cannot perform policy authorization. The error message naming GetPolicyEngine is misleading, and there is documented precedent in the same guide for these errors misattributing the gap.

Q5 — Is bedrock-agentcore.ap-south-1.amazonaws.com correct? Yes, that matches the documented format for a policy engine in ap-south-1.

Q6 — Is there an interim configuration that doesn't remove the ViaService or encryption-context restrictions? I'd reframe this. Add the two missing permissions first; if that unblocks creation, you never have to relax the KMS conditions and the question is moot. If it doesn't unblock, then temporarily dropping the kms:EncryptionContext StringEquals while keeping kms:ViaService is worth doing as a diagnostic, not as a configuration: if creation then succeeds, your encryption-context condition value is the problem and you've localised it in one attempt. Note that the docs' own key-policy examples use StringLike with arn:aws*:bedrock-agentcore:*:*:policy-engine/* rather than StringEquals on an exact ARN, so there is precedent for the looser form being what the service expects.

Suggested order of work

1. Simulate AuthorizeAction and PartiallyAuthorizeActions for both Gateway roles against the Policy Engine ARN and a Gateway ARN. Expect implicit deny — that's your confirmation before touching anything.
2. Add both actions to each Gateway execution role, with resources policy-engine/<exact-id> and gateway/*. Keep GetPolicyEngine scoped exactly as you have it.
3. Make sure the permissions boundary allows the two new actions on the Gateway ARN space, otherwise the boundary will silently cap them. Your boundary currently allows only the two read actions on Policy Engine ARN prefixes, so this needs a v6.
4. Run kms list-grants to confirm two grants per engine.
5. Retry the stack. If it still fails, run the encryption-context diagnostic from Q6.

What only Support can settle

If the above doesn't resolve it, the remaining unknowns are genuinely internal: the effective kms:ViaService value on the GenesisPolicyEngineCheck path, and whether that validation uses the assumed role directly or a grant. Neither is publicly documented at that level of detail, and CloudTrail deliberately doesn't expose the former. Your report is already in the right shape for a case — the request IDs plus the assumed-role session ARN plus the DryRun events are exactly what's needed. I'd add the list-grants output and the simulator results for the two additional actions, since those pre-empt the first two questions Support will ask.

Worth stating plainly: everything above about the required permission set and the grant-based KMS model is from current documentation and I've quoted it rather than paraphrased. The claim that the missing AuthorizeAction and PartiallyAuthorizeActions are the cause of your specific CreateGateway failure is inference from that documentation plus your evidence, not something I've reproduced. The step-1 simulation is cheap and will tell you whether the inference holds before you invest in a stack update.

u/Danilo's diagnosis is right on both counts. Just wanted to add a bit more on why each fix works.

For the missing permissions: the AgentCore IAM Permissions guide explicitly lists bedrock-agentcore:AuthorizeAction, PartiallyAuthorizeActions, and GetPolicyEngine as all three required on the execution role, and notes the InternalServerException symptom when they're absent. The resource scoping needs to cover both gateway/* and policy-engine/* in the same statement, not just one or the other. Exact ARN scoping fails here because the Gateway ARN doesn't exist yet at CreateGateway validation time, so the wildcard is the correct pattern.

For the KMS issue: the kms:ViaService condition on your execution role's identity policy won't fire because the Decrypt request arrives as a Forward Access Session from bedrock-agentcore.amazonaws.com, not directly from your role. The AWS KMS docs confirm kms:ViaService only applies for FAS calls. The fix is to move that Allow with the kms:ViaService condition onto the KMS key policy itself, with your execution role as the Principal.

The IAM simulator returning Allow here is a known limitation. It can't inject the FAS context keys (aws:CalledVia, kms:ViaService) that the live call chain produces, so the simulation result is genuinely misleading for this pattern. Your CloudTrail evidence showing the denial under session GenesisPolicyEngineCheck is the real ground truth.

One thing worth checking before reissuing the stack update: does your OctaDocAssuranceControlPlaneBoundary v5 permit-list AuthorizeAction and PartiallyAuthorizeActions? Adding them to the execution role policy but not the boundary will recreate the same failure.

I put together a short write-up on this exact fix pattern (corrected IAM JSON, KMS key policy statement, boundary amendment, and a CloudFormation patch to get from UPDATE_ROLLBACK_COMPLETE to UPDATE_COMPLETE). Happy to share it if that would help.

Agreed on the permissions gap and on the boundary point. Two corrections on the KMS reasoning, one of them to my own earlier answer.

kms:ViaService is valid in identity policies, so that isn't why the condition fails

The claim that the condition "won't fire because the Decrypt request arrives as a Forward Access Session ... not directly from your role" inverts the mechanism. From AWS KMS condition keys, the kms:ViaService entry is marked "Key policies and IAM policies", and the text states it plainly:

The kms:ViaService condition key is valid in IAM and key policy statements.

The same section also says:

This condition key only applies for Forward access sessions.

Both statements are true together, and the second is not a limitation on where the key can be used. It describes when the key is populated: the value exists only when a service makes the request on behalf of the principal. A FAS call is therefore the case in which a kms:ViaService condition in an identity policy is evaluated and can be satisfied. If the call were made directly by the role, the key would be absent and a StringEquals on it would fail to match.

So "the request arrives as FAS" is an argument that the identity-policy condition should work, not that it is inert. Moving the statement to the key policy may still be necessary — the key policy has to permit the operation too — but not for the reason given, and a reader could reasonably conclude from that rationale that their identity policy condition is doing nothing, which is not the case.

There is also a third authorization path in play that is neither of these. Per the AgentCore policy encryption guide, the service performs its cryptographic operations through two KMS grants created at policy-engine creation time. A grant is its own authorization mechanism; it is not an identity policy and not a key policy statement. That is why the list-grants check matters before rewriting either policy.

The discrepancy worth testing first, and a correction to my own answer

I told you bedrock-agentcore.ap-south-1.amazonaws.com is correct because it matches the format documented in the AgentCore guide. That is accurate as far as it goes, and I should have checked one more place before answering Q5 as cleanly as I did.

The KMS developer guide maintains a table titled "Services that support the kms:ViaService condition key," introduced as the AWS services "that are integrated with AWS KMS and support the use of the kms:ViaService condition key in customer managed keys." bedrock-agentcore does not appear in that table. The only Bedrock entry is:

Amazon Bedrock Model Copy | bedrock.AWS_region.amazonaws.com

I want to be careful about what this does and does not prove. It does not prove the condition key is unsupported on this path — that table has historically lagged behind service launches, and the AgentCore guide documents the value explicitly with a worked key-policy example, which is a strong counter-signal. What it does mean is that there are now two AWS documents that do not agree about whether bedrock-agentcore supports this condition key, and you have a denial whose mechanism would be fully explained if the value is not populated as the AgentCore guide implies.

That moves the priority order. A StringEquals condition on a context key that is absent from the request evaluates to no match, and the statement then contributes no Allow, producing exactly the "no identity-based policy allows it" wording you saw in CloudTrail. That is indistinguishable in the logs from having no statement at all.

So the diagnostic I suggested under your Q6 is now more than a fallback — I'd run it in parallel with the permission fix rather than after it. Concretely, on one role only, and temporarily:

• Keep kms:Decrypt on the exact key ARN.
• Keep the kms:EncryptionContext:aws:bedrock-agentcore-policy:policy-engine-arn condition, since your DryRun events confirm that value is present and matching.
• Remove only the kms:ViaService condition.

If creation then succeeds on that role and continues to fail on the untouched one, you have isolated it to that single condition key in one attempt, with the encryption-context restriction still in place. That is a useful answer either way, and it is the one piece of your Q1 that you can settle yourself without waiting on Support — you cannot read the value from CloudTrail, but you can determine whether a condition on it is satisfiable.

Add the outcome to your case. "A StringEquals on kms:ViaService for bedrock-agentcore.<region>.amazonaws.com prevents the internal validation from succeeding, while the documented encryption context matches" is a considerably sharper question than asking what the effective value is, and it also gives the service team a concrete documentation inconsistency to reconcile.

Answer (resolved)

AWS Support diagnosed this, and the diagnosis was correct. Posting the full resolution here because the failure surfaces one denial at a time and each one looks like an IAM mistake on your side.

Root cause

AgentCore's CreateGateway validation assumes the Gateway execution role using a plain sts:AssumeRole, not a forwarded access session. The probe runs under session names like GenesisPolicyEngineCheck and GenesisMCPTargetTargetEncryption.

Because it is not a FAS, kms:ViaService can never be present on those requests. The documented policy pattern for policy-engine encryption — kms:Decrypt conditioned on kms:ViaService = bedrock-agentcore.REGION plus an encryption context — therefore cannot authorize the very validation that CreateGateway performs. No amount of correcting the ViaService value helps; the condition key simply is not there. The IAM policy simulator says "allowed" because the simulator does not model the probe's request context.

The fix

Condition the key grant on the resource instead of the request. On the Gateway execution role:

- Sid: UseKeyForGatewayValidation
  Effect: Allow
  Action: [kms:Decrypt, kms:DescribeKey, kms:GenerateDataKey]
  Resource: <your CMK ARN>
  Condition:
    StringEquals:
      aws:ResourceTag/<your-tag-key>: <your-tag-value>

Resource tags are evaluated against the key itself, so they work regardless of how the caller assumed the role.

Three more gaps of the same class

The first fix unblocks the first probe and reveals the next. In order:

1. bedrock-agentcore:AuthorizeAction and PartiallyAuthorizeActions in the PERMISSIONS BOUNDARY of the Gateway role. The identity policy already granted them; the tell is the error phrasing — "no permissions boundary allows" rather than "no identity-based policy allows". Worth reading those two messages carefully, they point at different documents.
2. kms:GenerateDataKey, probed by a second session named GenesisMCPTargetTargetEncryption. Same plain-AssumeRole shape, so it goes in the same resource-tag-conditioned statement.
3. bedrock-agentcore:CreateWorkloadIdentity on the CLOUDFORMATION EXECUTION ROLE, targeting workload-identity-directory/default with no tags on the request. If that grant is conditioned on aws:RequestTag, it cannot match. Grant Create/DeleteWorkloadIdentity on the directory and .../workload-identity/* without request-tag conditions.

After all four, AWS::BedrockAgentCore::Gateway and GatewayTarget created normally, and the gateway now serves MCP initialize and tools/list through a CUSTOM_JWT authorizer with a Lambda interceptor.

A useful positive control

If you want to convince yourself the FAS explanation is real rather than a workaround: in the same account, against the same CMK, we have a Lambda that reads a DynamoDB table encrypted with that key. There kms:ViaService = dynamodb.REGION.amazonaws.com does match, and CloudTrail records the call with invokedBy: dynamodb.amazonaws.com — a genuine forwarded access session. Same key, same account: ViaService works on the DynamoDB path and cannot work on the Genesis probes. That contrast is the clearest way to tell the two situations apart.

Debugging tip

Denied KMS calls redact requestParameters in CloudTrail, so you cannot read the probe's conditions directly. What you can read is userIdentity.sessionContext for the session name, and invokedBy — which is exactly what distinguishes a FAS from a plain AssumeRole.