SSL issues in US + Canada with Certificate Manager and APIGateway

Question

I have a native mobile application on iOS and Android, that both connect to my backend services on AWS.

All of my DNS is handled by Route53 and all of my SSL is handled by AWS CertificateManager and the relevant service that SSL is used for.

On the AWS side, the clients connect to an APIGateway setup with a custom domain, which then backs off to a series of lambda functions.

I have always had a small portion of requests fail with an SSL error "An SSL error has occurred and a secure connection to the server cannot be made", but recently (specifically since October 25th) I have seen a huge increase in this in the US and Canada. Now roughly 35-40% of my users in the US and Canada get this error and are unable to connect to my backend, whereas in other areas (UK and Australia) it still remains low at ~10%.

![Ratio of unique users experiencing the issue from devices in the USA vs. UK](/media/postImages/original/IMddN9FctsQD69Bwv5oElZNg)
(Ratio of unique users experiencing the issue from devices in the USA vs. UK)

![Active users in the USA (blue) and users in USA experiencing the issue (red). User numbers have not increased, but error rate has.](/media/postImages/original/IMLQieR9gLRFix7l-BRqVMlQ)
(Active users in the USA (blue) and users in USA experiencing the issue (red). User numbers have not increased, but error rate has.)

This issue appears to effect both the iOS and Android apps which suggests it is not an issue at the client end, since they are both completely independent native apps with no shared code.

Any insight on what could be causing this, and more, what could have caused it to get so much worse recently in specific regions would be greatly appreciated, as I have been at a brick wall for some time now :|

re:Post Agent · Answer

Based on the information provided, it appears you're experiencing a significant increase in SSL errors for your mobile application users in the US and Canada, particularly since October 25th. This issue affects both iOS and Android apps, suggesting the problem is likely on the server-side rather than client-side. Here are some potential causes and steps to investigate:

1. Certificate issues: Check if your SSL certificate has expired or is nearing expiration. Even if it's still valid, there might be issues with the certificate chain or intermediate certificates.

2. API Gateway configuration: Verify that your custom domain setup in API Gateway is correct and that the SSL certificate is properly associated with it.

3. CloudFront distribution: For edge-optimized API endpoints, API Gateway sets up a CloudFront distribution. Ensure this distribution is functioning correctly and is properly bound to your ACM certificate.

4. Regional differences: The fact that the issue is more prevalent in the US and Canada could indicate a problem with specific edge locations or regional configurations. Check if there have been any recent changes or updates to your API Gateway or CloudFront settings that might affect these regions differently.

5. DNS issues: Since you're using Route 53, verify that your DNS records are correct and up-to-date, especially for the affected regions.

6. Certificate Authority Authorization (CAA) records: Check if there are any CAA records that might be blocking certificate issuance or renewal in the affected regions.

7. Recent changes: Review any changes made to your AWS infrastructure, especially around October 25th when the issue became more pronounced.

8. ACM service-linked role: Ensure that the ACM service-linked role (SLR) has the necessary permissions and is functioning correctly.

To troubleshoot:
1. Check the status of your certificate in the ACM console.
2. Verify the API Gateway custom domain configuration.
3. Review CloudWatch logs for API Gateway and related services for any error messages.
4. Test DNS resolution from various locations, especially in the affected regions.
5. Consider creating a new certificate and updating your API Gateway to use it, to rule out any issues with the current certificate.

If the issue persists after these checks, it may be worth contacting AWS Support for further assistance, as they can provide more detailed insights into your specific configuration and any potential regional issues.  
**Sources**  
[API Gateway problems - AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/troubleshoot-apigateway.html)  
[Troubleshoot other problems - AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/misc-problems.html)  
[Troubleshoot DNS validation problems - AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/troubleshooting-DNS-validation.html)  
[Troubleshoot certificate requests - AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/troubleshooting-cert-requests.html)

SSL issues in US + Canada with Certificate Manager and APIGateway

Relevant content