How do I troubleshoot HTTP 403 Forbidden errors from an API Gateway custom domain name that requires mutual TLS?

Short description

Note: API Gateway returns 403 Forbidden errors for a variety of reasons. This article addresses 403 Forbidden errors related only to mutual TLS. For information on troubleshooting other types of 403 Forbidden errors, see How do I troubleshoot HTTP 403 errors from API Gateway?

To invoke an API Gateway API with a custom domain name that requires mutual TLS, clients must present a trusted certificate in the API request. When a client invokes the API, API Gateway looks for the client certificate's issuer in your truststore.

The following conditions cause API Gateway to fail the TLS connection and return a 403 status code:

• API Gateway can't find the issuer of the client certificate in your truststore.
• The client certificate uses an insecure signature algorithm.
• The client certificate is self-signed.

If Amazon CloudWatch logging is activated for your API, then an error message that indicates the cause of the error appears in your execution logs.

Important: If the API request doesn't produce any CloudWatch Logs after logging is activated, then the 403 Forbidden error isn't related to mutual TLS.

For REST APIs

If you set up Amazon CloudWatch logging for your REST API, then one of the following error messages appears in your execution logs:

• Access denied. Reason: Could not find issuer for certificate
• Access denied. Reason: Client cert using an insecure Signature Algorithm
• Access denied. Reason: self signed certificate

For HTTP APIs

HTTP APIs don't support execution logging. To troubleshoot 403 Forbidden errors returned by a custom domain name that requires mutual TLS and invokes an HTTP API, complete the following steps:

1. Create a new API mapping for your custom domain name that invokes a REST API for test purposes only.
   Note: If you don't have a REST API to test, use the example PetStore REST API. Then, deploy the example API to a new stage, and create a new API mapping that uses your custom domain name.
2. Follow the instructions in the Resolution section of this article with the new API mapping that you created to your REST API.
3. Reroute the API mapping for your custom domain name back to your HTTP API.

Resolution

Confirm the cause of the error

1. Turn on CloudWatch logging for your REST API.

2. Configure execution and access logging. Note: When you configure access logging for this use case, use the following $context variables:

   { "accountId":"$context.accountId", "apiId":"$context.apiId", "domainName":"$context.domainName", "domainPrefix":"$context.domainPrefix", "error.message":"$context.error.message", "error.responseType":"$context.error.responseType", "extendedRequestId":"$context.extendedRequestId", "httpMethod":"$context.httpMethod", "identity.sourceIp":"$context.identity.sourceIp", "identity.clientCert.clientCertPem":"$context.identity.clientCert.clientCertPem", "identity.clientCert.subjectDN":"$context.identity.clientCert.subjectDN", "identity.clientCert.issuerDN":"$context.identity.clientCert.issuerDN", "identity.clientCert.serialNumber":"$context.identity.clientCert.serialNumber", "identity.clientCert.validity.notBefore":"$context.identity.clientCert.validity.notBefore", "identity.clientCert.validity.notAfter":"$context.identity.clientCert.validity.notAfter", "identity.userAgent":"$context.identity.userAgent", "path":"$context.path", "protocol":"$context.protocol", "requestId":"$context.requestId", "requestTime":"$context.requestTime", "requestTimeEpoch":"$context.requestTimeEpoch", "resourceId":"$context.resourceId", "resourcePath":"$context.resourcePath", "stage":"$context.stage", "responseLatency":"$context.responseLatency", "responseLength":"$context.responseLength", "status":"$context.status" }
   

   This access log template logs client certificate information when mutual TLS causes 403 errors. It also makes it easier to identify the caller that tried to invoke your API.

3. View your REST API's execution logs in CloudWatch to identify the cause of the error. If a 403 Forbidden error related to mutual TLS is logged, then you receive an error message similar to the following:

   Extended Request Id: {extendedRequestId} Access denied. Reason: {reason}
   ForbiddenException Forbidden: {requestId}
   

Resolve "Access denied. Reason: Could not find issuer for certificate" errors

Verify that the issuer of the client certificate in the API request is included in the custom domain name's truststore

The full certificate chain of the client certificate (client.pem) in the API request must be included in your custom domain name's truststore. The truststore (bundle.pem) is a file stored in Amazon Simple Storage Service (Amazon S3). To verify if the issuer of the client certificate is included in the required truststore, run the following OpenSSL command:

openssl verify -CAfile bundle.pem client.pem

If the certificate bundle contains intermediate certificate authorities, then run the following OpenSSL command:

openssl verify -CAfile rootCA.pem -untrusted intCA.pem client.pem

If the issuer of the client certificate in the API request is included in the required truststore, then the command returns an OK response.

If the issuer of the client certificate isn't included in the required truststore, then the command returns the following error:

error X at Y depth lookup: unable to get local issuer certificate

To resolve this error, upload a new truststore to Amazon S3 that includes the full certificate chain for the client certificate.

Verify that all of the client certificates in your custom domain name's truststore are valid

If one of the client certificates in your custom domain name's truststore isn't valid, then some clients might not be able to access your API. To confirm that the client certificates in your truststore are valid, complete the following steps:

1. Open the Amazon API Gateway console.

2. In the navigation pane, choose Custom domain names. Then, choose your custom domain name that requires mutual TLS.

3. In the Domain details section, check for a warning such as: Your truststore bundle has 1 invalid certificates.

4. If you see the warning, decode the certificates in your truststore to identify which certificate produced the warning. The following OpenSSL command displays the subject and contents of a certificate:

   openssl x509 -in certificate.crt -text -noout
   

5. Update or remove the certificates that produced the warning. Then, upload a new truststore to Amazon S3.

For more information, see Troubleshooting certificate warnings.

Note: If their certificate chain is preserved, then API Gateway accepts client certificates signed directly by the root certificate authority or any other intermediate certificate authority. To validate client certificates signed by only the last intermediate certificate authority, use a request parameter-based AWS Lambda authorizer. You can create a custom validation algorithm in the request-based Lambda authorizer. The client certificate can be found in the input from the API request under requestContext.identity.clienCert.

Resolve "Access denied. Reason: Client cert using an insecure Signature Algorithm" errors

Verify that your truststore text file uses a supported hashing algorithm. API Gateway supports the following hashing algorithms in the truststore:

• SHA-256 or stronger
• RSA-2048 or stronger
• ECDSA-256 or stronger

To confirm your truststore text file uses a supported hashing algorithm, run the following OpenSSL command:

openssl x509 -in client.crt -text -noout | grep 'Signature Algorithm'

The command response returns your truststore's signature algorithm. If the algorithm isn't supported, update your client certificate to use a supported algorithm. Then, upload a new truststore to Amazon S3.

For more information, see Configuring your truststore.

Resolve "Access denied. Reason: self signed certificate" errors

Verify that the self-signed client certificate in the API request isn't altered or corrupted. The client certificate signing request (my_client.csr), client certificate private key (my_client.key), and the client certificate public key (my_client.pem) must match.

To compare moduli, run the following OpenSSL commands:

openssl req -noout -modulus -in my_client.csr
openssl rsa -noout -modulus -in my_client.key
openssl x509 -noout -modulus -in my_client.pem

Note: To produce a shorter hash value for easier comparison, use a pipe on the output modulus. See the following openssl sha1 example:

$ openssl [operation] -noout -modulus -in [data] | openssl sha1

A valid output example looks similar to the following:

2143831a73a8bb28467860df18550c696c03fbcb
2143831a73a8bb28467860df18550c696c03fbcb
2143831a73a8bb28467860df18550c696c03fbcb

To confirm data integrity, verify that there isn't any data modification at the content level. Run the following diff command:

diff client.crt bundle.crt

If the moduli don't match or the diff command returns differences, regenerate the client certificate and update your truststore. Then, upload a new truststore to Amazon S3.

For more information, see Configuring your truststore.

Related information

How to turn on mutual TLS authentication for your REST APIs in API Gateway