Issue to connect to Postgres RDS using AWS Glue

0

Hello, I'm trying to set up a Lake Formation to keep some data from RDS. Using AWS documentation and tutorial I created:

  1. AWS user with administrative permissions, AmazonAthenaFullAccess, AWSGlueConsoleFullAccess, and AWSLakeFormationDataAdmin.
  2. Lake formation
  3. AWS Glue connection to RDS using JDBC driver, VPC and RDS security group After that I tried to Test that connection, use it in Blueprint, and Crawler. Unfortunately, every time, after several minutes of work, it is Fail. The error is : Getting error: [1308dfbe-8bf9-494c-aa47-b57b07ec3713] ERROR : Internal Service Exception. Any ideas how to fix that issue?
2 Answers
0

Hello Andrey,

error description is not that specific. You could try some ways to resolve issue such as-

  • Check if RDS security group allows ingress traffic on the database port (e.g 3306) from the Glue VPC security group. Glue needs to connect to RDS from there.

  • Verify the RDS endpoint, database name, username, and password are all correct in the Glue connection. Test the connection parameters directly from a client like SQL Workbench.

  • Recheck networking/routing between the Glue VPC, RDS subnet groups. Ensure there is a VPC peering connection or routes set up.

  • increase the timeout for the Glue connection test. The default can sometimes be too short with networking delays. Try say 3 -4 mins.

  • Check CloudWatch Logs for specific log failures from Glue during the connection test. Details could help point to root causes.

  • Try creating a simpler Glue table/database pointing to RDS to test first. this will rule out other moving parts such as crawlers etc.

  • Check for service limits in Glue and Rds

Hope this helps, pls let me know if works out. Thanks

AWS
EXPERT
Ben Lee
answered 6 months ago
  • Hi Ben, Thank you for your reply.

    1. My RDS security group allows All Traffic inbound and outbound
    2. Everything RDS endpoint, database name, username, and password are all correct in the Glue connection are correct. No issues to connect to it from EC2 or Postgres DB Gui tools
    3. Glue connection uses the same VPC and subnet group as RDS
    4. This is not a timeout issue because connection test ran, at least 3 min before Fail
    5. The message, ERROR : Internal Service Exception, is from CloudWatch
    6. I created a new RDS with one table( 4 rows) and tried to connect it to the Glue. Result is the same
    7. How can I check for service limits? Thanks again for your reply
0

Ensure the glue connection is on a private subnet with either a route to a NAT gateway or you have setup the glue vpc endpoint.

This may resolve your issue.

profile picture
EXPERT
answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions