How do I troubleshoot 500 errors when I use VPC link integrations with API Gateway REST APIs?

Short description

API Gateway with VPC link integrations can fail to forward traffic to backend resources and return an HTTP 500 status code for the following reasons:

• The VPC link is in the Failed state. Or, the endpoint service is deleted or in the Rejected state.
• The registered target for the target group is unhealthy, or isn't in use.
• Security groups don't allow traffic on specific ports.
• The network access control list blocks the traffic.
• The target isn't listening on the target ports.
• The domain name certificate doesn't match the TLS-activated Network Load Balancer or Application Load Balancer.

If Amazon CloudWatch Logs is activated for your API, then an error message that has the cause of the error appears in the execution logs.

Example HTTP 500 error messages in CloudWatch logs

HTTP 500 error messages in your CloudWatch logs are similar to the following examples:

"Thu May 04 17:48:21 UTC 2024 : Execution failed due to configuration error: There was an internal error while executing your request Thu May 04 17:48:21 UTC 2024 : Method completed with status: 500"

-or-

"Thu May 04 19:50:21 UTC 2024 : Execution failed due to configuration error: Host name 'domain.com.com' does not match the certificate subject provided by the peer (CN=myinstance.com)
Thu May 04 19:50:21 UTC 2024 : Method completed with status: 500"

Resolution

Confirm the cause of the error

Turn on CloudWatch API logging. Make sure that you configure execution logging.

Note: When you configure the logging settings from the dropdown list, choose Full request and response logs.

To identify the cause of the errors, review the REST APIs execution logs in CloudWatch.

Check for the error message in the execution logs

Error: Execution failed due to configuration error: There was an internal error while executing your request

Check the following configurations:

• Verify that the VPC link load balancer exists and wasn't deleted.
• Verify that the VPC link is in the AVAILABLE state. If the VPC link is in the FAILED state, then you must create a new VPC link and associate the link with your API.
  Note: Deploy the API after you modify the integration request.
• Verify the following:
  The VPC link endpoint connection associated to your target Network Load Balancer is in the AVAILABLE state.
  The region-specific API Gateway account ID is listed in Allow principals.
  The Endpoint Service's Endpoint connections lists an endpoint connection from the mentioned account ID in the Available State.
• If the VPC ID is referenced with a stage variable, then verify that the VPC link ID is correct.
• Verify that the VPC link load balancer listens on the HTTP/HTTPS port that the request is configured for. Check that the listeners are configured for the correct port. Make sure that the network access control lists (network ACLs) don't block the request.
• Verify that the target group accepts the request. The network ACLs must allow inbound and outbound traffic, and the security groups must allow inbound traffic for the configured ports.
• If the request returns a 500 error, then the connection might receive TCP total number of reset (RST) packets. Verify that the backend server is running. Verify that there's a service running on the backend target of the target port. Verify that the backend target listens on the target port.
• Verify that the Enforce inbound rules on AWS PrivateLink traffic option is turned off. Enforce inbound rules on PrivateLink traffic applies all inbound and outbound security group rules to traffic from the VPC link. If you don't configure the security group to allow traffic from the VPC link, then the security group can block API Gateway traffic. To apply security group rules to traffic that doesn't come from the VPC link, turn off Enforce inbound rules on PrivateLink traffic.

Error: Execution failed due to configuration error: Host name 'domain.com.com' does not match the certificate subject provided by the peer (CN=myinstance.com)

Make sure that the endpoint domain name matches the certificate that the TLS-activated load balancer target returns. Certificates that you configure at the target instance must be trusted from the supported certificate authority provider list.

Error: Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This error occurs when the certificate returned by the integration is flagged as not valid by the REST API service. Even when insecureSkipVerification is set to true on the integrations TlsConfig property, API Gateway performs basic certificate validation. The validation reviews the following information:

• The certificates expiration date
• The hostname
• The presence of a root certificate authority (CA)
  Note: The complete certificate chain must be included and returned by the integration for API Gateway to verify the trust chain and establish a secure connection. The complete chain includes all intermediate certificates from the leaf (server) certificate up to the root CA certificate.

Error: Execution failed due to configuration error: Cannot verify ECDH ServerKeyExchange signature

This error occurs when a key and the corresponding certificate don't match, and the TLS handshake fails. To resolve this issue, check the contents of the files that you used for your configured certificate authorities, certificates, and keys.

Error: Execution failed due to an internal error

This error occurs when API Gateway doesn't connect to the load balancer over the VPC link because the target resets on the load balancer. To resolve this issue, set a longer timeout on the target than the default timeout (350 seconds) that's on the load balancer.

Error: Execution failed due to a VPC link error

API Gateway is a distributed service that has many dependencies. You might see errors when any of these dependencies have transient connection failures. For low-frequency errors, it's a best practice to implement retries with exponential backoff. If the frequency of errors is high, then contact AWS Support. Make sure to provide the API Gateway execution logs that indicate these errors.

Error: Execution failed due to configuration error: Not a valid protocol version: {Text of string}

This error occurs when the integration responded with an HTTP response that's not valid and doesn't adhere to the HTTP specification. This might mean that the integrated backend is sending back data that isn't valid to API Gateway.

To troubleshoot this error, take the following actions:

• Reproduce the request that you made from API Gateway. To test the service endpoint that API Gateway uses, create a VPC endpoint that's linked to the service endpoint ID. You can also run packet captures to review the response that's returned from the integrated backend.
• To avoid duplicate encryption, change the load balancer protocol to TLS or TCP, depending on the response of the targets.
• Make sure that insecureSkipVerification is set to true on the integrations TlsConfig property.

Related information

Understanding VPC links in Amazon API Gateway private integrations

Monitor REST API execution with Amazon CloudWatch metrics

Set up a Network Load Balancer for API Gateway private integrations