ECS EC2 and Postgres container connection by Service Connect
I have ECS cluster with EC2 instances. My purpose is to deploy Postgres and Backend services so backend
could connect to postgres
. I don't want to use RDS for developing purposes, so I've created both services involving AWS Service Connect. I don't want to use awsvpc networking, because instances has ENI limits, so all my task definitions have bridge
networkMode. But my problem is backend cann't connect to postgres via DNS name, despite I allowed all connections in postgres container (POSTGRES_HOST_AUTH_METHOD=trust
)
For debugging purposes I deployed alpine
container to ping both postgres
and backend
. Service Connect
has the same namespace for all 3 services and has next configuration:
Postgres: Client and Server | port name: postgres-5432 | discovery: postgres | DNS: postgres.inbalance.develop | port: 5432
Backend: Client and Server | port name: connector-9090 | discovery: connector | DNS: connector.inbalance.develop | port: 9090
Alpine: Client only
Task definitions applied below
After some manual researches (connection to EC2 instance directly) I've discovered that I can connect from Alpine to Backend:\
curl -i connector.inbalance.develop:9090/ping
HTTP/1.1 200 OK
content-type: application/json
date: Thu, 22 Aug 2024 20:02:29 GMT
content-length: 18
x-envoy-upstream-service-time: 18
server: envoy
{"message":"pong"}
and I tried to check connection from Alpine to Postgres:\
curl -i postgres.inbalance.develop:5432
HTTP/1.1 503 Service Unavailable
content-length: 118
content-type: text/plain
date: Thu, 22 Aug 2024 20:04:11 GMT
server: envoy
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection termination
I know it's useless as postgres doesn't use HTTP protocol, but at least I found that something rejects connection.
Then I tried psql
with 2 ssl modes - disable
and require
. It doesn't tell me much, but here are results:
psql postgresql://postgres:password@postgres.inbalance.develop:5432/inbalance?sslmode=disable
psql: error: connection to server at "postgres.inbalance.develop" (127.255.0.2), port 5432 failed: expected authentication request from server, but received H
psql postgresql://postgres:password@postgres.inbalance.develop:5432/inbalance?sslmode=require
psql: error: connection to server at "postgres.inbalance.develop" (127.255.0.2), port 5432 failed: received invalid response to SSL negotiation: H
And the most interesting part - I can connect using instance's private IP address.
psql postgresql://postgres:password@172.31.15.199:5432/inbalance?sslmode=disable
psql (16.3, server 16.4 (Debian 16.4-1.pgdg120+1))
Type "help" for help.
inbalance=#
ACLs and security groups are configured correctly (with full access).
Has somebody encountered with this problem? Is this issue related to incorrect Service Connect setup, wrong Postgres configuration or my misunderstanding of Service Connect?
PostgresTaskDefinition:
{
"family": "EC2PostgresDevelop",
"containerDefinitions": [
{
"name": "postgres",
"image": "public.ecr.aws/docker/library/postgres:16.4",
"cpu": 0,
"portMappings": [
{
"name": "postgres-5432",
"containerPort": 5432,
"hostPort": 5432,
"protocol": "tcp"
}
],
"essential": true,
"command": [
"postgres",
"-c",
"log_statement=all",
"-c",
"log_connections=true",
"-c",
"log_min_messages=DEBUG1",
"-c",
"log_min_error_statement=DEBUG1",
"-c",
"client_min_messages=DEBUG1"
],
"environment": [
{
"name": "POSTGRES_USER",
"value": "postgres"
},
{
"name": "POSTGRES_PASSWORD",
"value": "password"
},
{
"name": "POSTGRES_DB",
"value": "inbalance"
},
{
"name": "POSTGRES_HOST_AUTH_METHOD",
"value": "trust"
}
],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"disableNetworking": false,
"readonlyRootFilesystem": false,
"extraHosts": [],
"ulimits": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/PostgresDevelop",
"mode": "non-blocking",
"awslogs-create-group": "true",
"max-buffer-size": "25m",
"awslogs-region": "eu-north-1",
"awslogs-stream-prefix": "ecs"
},
"secretOptions": []
},
"healthCheck": {
"command": [
"CMD-SHELL",
"pg_isready -U postgres -d inbalance"
],
"interval": 20,
"timeout": 3,
"retries": 3,
"startPeriod": 20
},
"systemControls": []
}
],
"taskRoleArn": "arn:aws:iam::*:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::*:role/ecsTaskExecutionRole",
"networkMode": "bridge",
"volumes": [
{
"name": "PostgresDevelopVolume",
"host": {
"sourcePath": "/var/lib/postgresql/data/"
}
}
],
"requiresCompatibilities": [
"EC2"
],
"cpu": "512",
"memory": "512",
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
}
}
BackendTaskDefinition
{
"family": "Connector",
"containerDefinitions": [
{
"name": "connector",
"image": "*.dkr.ecr.eu-north-1.amazonaws.com/connector:2",
"cpu": 0,
"portMappings": [
{
"name": "connector-9090",
"containerPort": 9090,
"hostPort": 0,
"protocol": "tcp",
"appProtocol": "http"
}
],
"essential": true,
"environment": [],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"ulimits": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/Connector",
"mode": "non-blocking",
"awslogs-create-group": "true",
"max-buffer-size": "25m",
"awslogs-region": "eu-north-1",
"awslogs-stream-prefix": "ecs"
},
"secretOptions": []
},
"healthCheck": {
"command": [
"CMD-SHELL",
"wget --quiet --tries=1 --spider http://localhost:9090/health || exit 1"
],
"interval": 5,
"timeout": 2,
"retries": 3,
"startPeriod": 5
},
"systemControls": []
}
],
"taskRoleArn": "arn:aws:iam::*:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::*:role/ecsTaskExecutionRole",
"networkMode": "bridge",
"requiresCompatibilities": [
"EC2"
],
"cpu": "256",
"memory": "256",
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
}
}
- Newest
- Most votes
- Most comments
It's resolved automatically, perhaps CloudMap DNS caching for postgres.inbalance.develop
.
Relevant content
- asked a year agolg...
- asked 2 years agolg...
- asked 2 years agolg...
- asked 6 months agolg...
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated 4 months ago