How do I troubleshoot 500 errors when I use VPC link integrations with API Gateway REST APIs?

Short description

API Gateway with VPC link integrations might fail to forward traffic to backend resources.

It might also return an "HTTP 500" status code for one of the following reasons:

• The VPC link is in the Failed state. Or, the endpoint service doesn't exist or is in the Rejected state.
• The target group shows the registered target as unhealthy or not in use.
• Security groups don't allow traffic on specific ports.
• The network access control list (network ACL) blocks the traffic.
• The target doesn't listen on the target ports.
• The domain name certificate doesn't match the TLS-activated Network Load Balancer or Application Load Balancer.

If you activated Amazon CloudWatch Logs for your API, then an error message that has the cause of the error appears in the execution logs.

Resolution

Validate backend connectivity

Before you troubleshoot API Gateway, take one of the following actions:

• Use tools, such as curl or Postman, to send a request to the Network Load Balancer DNS name from within your VPC. If the request fails, then there's an issue with your Network Load Balancer configuration, backend target health, or networking.
• Use VPC Reachability Analyzer to validate traffic flow from the API Gateway VPC link subnets to your backend resources.

Confirm the cause of the error

Activate CloudWatch API logging. Also, configure execution logging.

Note: When you configure the logging settings, set the logging level to Error and info logs and select Data tracing. These settings provide you with the full request and response logs.

To identify the cause of the errors, review the REST APIs execution logs in CloudWatch. Then, troubleshoot the issue based on the error message that you receive.

Troubleshoot "There was an internal error while executing your request"

You receive the following error message:

"Error: Execution failed due to configuration error: There was an internal error while executing your request"

To resolve this issue, check the following configurations:

• The VPC link load balancer exists and wasn't deleted.
• The VPC link is in the Available state. If the VPC link is in the Failed state, then you must create a new VPC link and associate the link with your API.
  Note: Deploy the API after you modify the integration request.
• The VPC link endpoint connection that's associated to your target Network Load Balancer is in the Available state.
• You listed the AWS Region-specific API Gateway account ID in Allow principals.
• An endpoint connection from the API Gateway account in the endpoint service's Endpoint connections is in the Available State.
• If API Gateway references the VPC ID with a stage variable, then verify that the VPC link ID is correct.
• The VPC link load balancer listens on the HTTP/HTTPS port that you configured the request for. Make sure that you configured the listeners for the correct port, and the network ACLs don't block the request.
• The target group accepts the request. The network ACLs must allow inbound and outbound traffic, and the security groups must allow inbound traffic for the configured ports.
• If the request returns an "HTTP 500" status code error, then the connection might receive TCP total number of reset (RST) packets. The backend server must be running. Also, a service must be running on the backend target of the target port. Verify that the backend target listens on the target port.
• You deactivated inbound rules for AWS PrivateLink traffic. The Enforce inbound rules on PrivateLink traffic option applies all inbound and outbound security group rules to traffic from the VPC link. If you don't configure the security group to allow traffic from the VPC link, then the security group might block API Gateway traffic.

Troubleshoot "Host name 'domain.com.com' does not match the certificate subject provided by the peer"

You receive the following error message:

"Error: Execution failed due to configuration error: Host name 'domain.com.com' does not match the certificate subject provided by the peer (CN=myinstance.com)"

Make sure that the endpoint domain name matches the certificate that the TLS-activated load balancer target returns. The supported certificate authority provider list must trust the certificates that you configure at the target instance.

Troubleshoot "PKIX path building failed:...unable to find valid certification path to requested target"

You receive the following error message:

"Error: Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"

This error occurs when the REST API service flags a certificate that the integration returns as not valid. Even when you set insecureSkipVerification to true on the integration's TlsConfig property, API Gateway performs basic certificate validation.

API Gateway reviews the following information:

• The certificate's expiration date
• The hostname
• The presence of a root certificate authority (CA)
  Note: The integration must include and return the complete certificate chain for API Gateway to verify the trust chain and establish a secure connection. The complete chain includes all intermediate certificates from the server certificate up to the root CA certificate. Also, the root CA must include keyUsage with keyCertSign and basicConstraints with CA:TRUE. For more information, see x-amazon-apigateway-integration.tlsConfig object.

Troubleshoot "Cannot verify ECDH ServerKeyExchange signature"

You receive the following error message:

"Error: Execution failed due to configuration error: Cannot verify ECDH ServerKeyExchange signature"

This error occurs when a key and the corresponding certificate don't match, and the TLS handshake fails. To resolve this issue, check the contents of the files that you used for your configured CAs, certificates, and keys.

Troubleshoot "Execution failed due to an internal error"

You receive the following error message:

"Error: Execution failed due to an internal error"

This error occurs when API Gateway doesn't connect to the load balancer over the VPC link because the target resets on the load balancer. To resolve this issue, set a longer timeout on the target than the load balancer's default timeout of 350 seconds.

Troubleshoot "Execution failed due to a VPC link error"

You receive the following error message:

"Error: Execution failed due to a VPC link error"

API Gateway has many dependencies. You might see errors when one of the dependencies has transient connection failures. For low-frequency errors, it's a best practice to implement retries with exponential backoff. If the frequency of errors is high, then contact AWS Support. Make sure that you provide the API Gateway execution logs that show the errors.

Troubleshoot "Execution failed due to configuration error: Not a valid protocol version"

You receive the following error message:

"Error: Execution failed due to configuration error: Not a valid protocol version"

This error occurs when the integration responds with an HTTP response that isn't valid and doesn't adhere to the HTTP specification. The integrated backend might send back data that isn't valid to API Gateway.

To troubleshoot this issue, take the following actions:

• Reproduce the request that you made from API Gateway. To test the service endpoint that API Gateway uses, create a VPC endpoint that's linked to the service endpoint ID. You can also run packet captures to review the response that the integrated backend returns.
• To avoid duplicate encryption, change the load balancer protocol to TLS or TCP, depending on the response of the targets.
• Make sure that insecureSkipVerification is set to true on the integration's TlsConfig property.

Monitor and detect errors and issues

Use the following tools to determine whether the issue is transient or there are persistent configuration issues:

• Monitor 5XXError and IntegrationLatency API Gateway metrics in CloudWatch to detect patterns.
• Activate AWS X-Ray tracing to monitor request paths from API Gateway to the VPC Link, the VPC Link to the Network Load Balancer, and the Network Load Balancer to the backend.

Related information

Understanding VPC links in Amazon API Gateway private integrations

Monitor REST API execution with Amazon CloudWatch metrics

Set up a Network Load Balancer for API Gateway private integrations