I encountered the same problem. Used your solution, thanks @bansalakhil. The only one thing is that I didn't change the password_encryption globally. I did it for the session.

set password_encryption = 'md5';   
CREATE USER "read_only_user" WITH ENCRYPTED PASSWORD 'my_super_password';  
GRANT pg_read_all_data TO read_only_user;

I was able to use the correct driver.

1. Glue seems to use a postgresql driver incompatible with Postgresql14 by default

2. Download the latest version from https://jdbc.postgresql.org/download/ and make the .jar available on s3

3. The custom driver .jar uploaded to a s3 bucket will only be used if the JDBC driver class name is overridden

   • So make sure the Jdbc Driver class name is not empty , but it contains the default value: org.postgresql.Driver

Testing connections is not (yet) supported, this will throw the following exception:

com.amazonaws.glue.jobexecutor.commands.exception.CommandExecutorException: com.amazonaws.services.glue.exceptions.InvalidInputException: Testing connections with custom drivers is not currently supported.

Data crawling works for me now. I assume other Glue jobs will work accordingly.

I met same issue and originally asked at Stack Overflow. Later I found this thread and thanks!

I posted what I found in Stack Overflow and here is a copy:

After more research, I found the issue is because AWS Glue is using an old JDBC driver. So I have to upload the latest driver manually to use.

Step 1. Download latest JDBC driver and upload to S3

Download latest JDBC driver at https://jdbc.postgresql.org/download/ As of today, I downloaded latest JDBC driver 42.6.0 Java 8 version which filename is postgresql-42.6.0.jar.

Then I uploaded at s3://my-bucket/aws-glue/postgresql-jdbc-driver/postgresql-42.6.0.jar

Step 2. Update the JDBC connection

Update your JDBC connection in AWS Glue -> Connectors with

• JDBC Driver Class name: org.postgresql.Driver
• JDBC Driver S3 Path: s3://my-bucket/aws-glue/postgresql-jdbc-driver/postgresql-42.6.0.jar

Step 3. Add new access for the AWS Glue IAM role that is being used

In order to use my own JDBC driver, I need add the following permissions to the IAM role:

• Grant permissions for the following job actions: CreateJob, DeleteJob, GetJob, GetJobRun, StartJobRun.
• Grant permissions for IAM actions: iam:PassRole.
• Grant permissions for Amazon S3 actions: s3:DeleteObjects, s3:GetObject, s3:ListBucket, s3:PutObject.
• Grant service principal access to bucketin the IAM policy and your JDBC driver S3 location.

The default AWSGlueServiceRole policy has CreateJob, DeleteJob, GetJob, GetJobRun, StartJobRun. So I just need provide the rest of access to a new policy AWSGlueServiceRole-MyPolicy I created:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"s3:GetObject",
				"s3:PutObject",
				"s3:ListBucket",
				"s3:DeleteObject"
			],
			"Resource": [
                "arn:aws:s3:::my-bucket/aws-glue/postgresql-jdbc-driver/postgresql-42.6.0.jar",
                "arn:aws:s3:::my-bucket"
			]
		},
		{
			"Effect": "Allow",
			"Action": [
				"iam:PassRole"
			],
			"Resource": [
				"arn:aws:iam::************:role/service-role/AWSGlueServiceRole-***"
			]
		}
	]
}

Now the IAM role should have

• The default AWSGlueServiceRole policy
• My new policy AWSGlueServiceRole-MyPolicy

Note the "Test Connection" will still fail with error log

Caused by: com.amazonaws.services.glue.exceptions.InvalidInputException: Testing connections with custom drivers is not currently supported. Caused by: com.amazonaws.services.glue.exceptions.InvalidInputException: Testing connections with custom drivers is not currently supported.

However, the crawler should succeed!

Extra Notes

Based on AWS Glue doc, Glue 4.0 is using PostgreSQL JDBC driver 42.3.6 which is quite new. However, I am not sure why the connection still failed.

Note my Amazon RDS is using Postgres 15 this case, and my show password_encryption; returns scram-sha-256. I do not need set password_encryption = 'md5'; like some users did.

Reference

• https://repost.aws/questions/QUpkrhcfkYQtS2adbjpQ7quQ/cannot-connect-from-glue-to-rds-postgres#AN2SxYJ-3pT425XOS0GhUmQg

I am using RDS with Postgres 15, and stuck with the same error.

Tried the suggestion given by Jeroen, by uploading a custom jar file s3 and other related steps mentioned above. Even then the crawler failed after running for 8 minutes. Error logged was: ERROR : Internal Service Exception

I got the same error with RDS PostgreSQL engine version 14.6. But it successfully connected when I changed RDS PostgreSQL engine version to 13.10.

In CloudWatch go to Log groups and check if you have a log group with a name like /aws-glue/testconnection/output/<connectionName> that might give some more information. (The link in testconnection sends you to /aws-glue/testconnection/error/<connectionName>)

We had the same error in the error logs, however we did have information in the output logs of the connection tests.

For us that gave a slightly clearer error message that we solved by changing our Database authentication method from "Password and IAM database authentication" to "Password authentication"

Seeing this issue as well.

Looking into the output logs like @M_Mooiweer recommended provided this additional insight.

Check that your connection definition references your JDBC database with correct URL syntax, username, and password. The authentication type 10 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it is using an authentication scheme supported by the driver

Some other posts indicate that this may be related to scram vs md5 password encryption between Postgresql 13 and 14 so I explored this a bit further.

The JDBC Driver version 42.2.0+ should support scram. Per this doc, Glue 3.0-and-up support compatible drivers, but I'm not sure which version of Glue, if any, would be used when testing a connection.

Uploading a .jar of a known-compatible driver (works locally) to S3 and specifying it in the connection also does not appear to change the outcome.

Other folks have mentioned downgrading from postgresql 14 to 13, but I'm unsure if that's a sustainable solution in all cases.

Starting from Postgres version 14 scram-sha-256 is the default password encryption type. And it looks like it is not being supported by the JDBC driver used by Glue.

This is how I solved it in my case.

1. Create a new parameter group if using the default parameter group, and attach to RDS
2. Changed password_encryption to md5
3. RDS reboot may be required
4. Connect to DB using existing credentials, this must be using scram-sha-256
5. Create a new user and grant access to the required db/schema:

CREATE ROLE glue_readaccess;
GRANT CONNECT ON DATABASE MY_DB TO glue_readaccess;
GRANT USAGE ON SCHEMA public TO glue_readaccess;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO glue_readaccess;
CREATE USER aws_glue WITH PASSWORD '323233232klfdjkfdjkfjkdjfjdfjdkjfdjk';
GRANT glue_readaccess TO aws_glue;

6. Check that new user has encryption_type to md5, older user should have scram-sha-256. At this moment both types of users should be able to connect to DB with their respective credentials.
7. Note: From now onwards all new users would have md5 password encryption. Even if we reset the password for old users, those also are md5
8. Use this new user aws_glue in your Glue connection. This should work. It worked in my case.

Hi,

I have the same issue here. I several glue connections from RDS Postgres in the past, I have never faced any issue. Today, I tried to create a connection, I get the same error message.