SSL Handshake Failures with RDS Proxy and PostgreSQL in NestJS Application

Better check on below first:

1. Verify SSL Certificates:

• Ensure that your application is using the correct SSL certificate for RDS Proxy. AWS provides a root CA certificate that you can download and configure in your application.

• If you're encountering errors like "self-signed certificate in certificate chain," you may need to explicitly trust the AWS RDS CA certificate in your NestJS application.

2. Update Connection Settings:

• In your NestJS application, ensure that the database connection settings include the ssl option. For example, if you're using TypeORM:

TypeOrmModule.forRoot({
  type: 'postgres',
  host: process.env.DB_HOST,
  port: parseInt(process.env.DB_PORT, 10),
  username: process.env.DB_USER,
  password: process.env.DB_PASS,
  database: process.env.DB_NAME,
  ssl: {
    rejectUnauthorized: true,
    ca: fs.readFileSync('path/to/rds-ca.pem').toString(),
  },
});

• Replace path/to/rds-ca.pem with the actual path to your downloaded CA certificate.

3. Check TLS Versions:

• Ensure that your RDS Proxy and PostgreSQL instance are configured to use compatible TLS versions. AWS RDS supports TLS 1.2 and 1.3, so make sure your application is not attempting to use an older version.

4. Connection Pooling:

• RDS Proxy manages connection pooling, but you should ensure that your application is not creating excessive connections. Adjust the pool settings in your database configuration to align with RDS Proxy's capabilities.

5. Debugging with OpenSSL:

• Use the openssl command to test the SSL connection to your RDS Proxy:

openssl s_client -connect <rds-proxy-endpoint>:5432 -showcerts

This can help identify issues with the certificate chain or TLS handshake.

6. Review Logs:

Check the logs in your NestJS application and RDS Proxy for detailed error messages. These logs can provide clues about the root cause of the handshake failures.

I doubt that RDS proxy has a bug.

--

TCP Connection

Connection established through 3-way handshake (SYN, SYN-ACK, ACK)

TLS Connection

Security protocol operating on top of TCP that negotiates encryption parameters, key exchange, authentication, etc. Managed by Session ID

Estimated RDS Proxy Mechanism

Maintains TCP connections using keepalive mechanism while managing TLS sessions separately

Potential Problem Scenarios

Client attempts communication with incorrect TLS session information:

TLS session information cached by the client may have already expired or been removed by the proxy This can result in the error: "Client network socket disconnected before secure TLS connection was established"

Abnormal connection termination handling:

Client sends TLS "close_notify" but RDS Proxy fails to process it properly TCP connection is still considered active while terminated at the TLS level This can result in the error: "Connection terminated unexpectedly"