NGINX proxy on ECS fails with 502 bad Gateway

Question

Hi,

I have managed to deploy a reverse proxy onto ECS as a service, which fails to bind on the TCP 8999 a few times, but it seems to be listening already. I am also observing some weird issues trying to proxy the request to the authentication service on ECS which fails with 502 Bad Gateway (a direct CURL to `http://authentication.ecs-demo.test:8083/foo` succeeds). I have ran `nginx -t` and there were no issues. I am clueless at what could be the issue. Both access log and error log are redirected to `/dev/stdout` and `/dev/stderr` but when I tail them nothing comes in. I am clueless as to what could be the problem. Below is the snippet of a valid nginx.conf file...

Thanks for the help.
Cheers.

```
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
}

http {
       ...
	gzip on;
	gzip_proxied any;
	gzip_types text/plain application/json;
	gzip_min_length 1000;

server {
		listen 8999;
		server_name _;

# This max value should not be smaller than the limit set in the UI for document uploading.
		client_max_body_size 16M;

location /foo {
			set $auth http://authentication.ecs-demo.test:8083;
			proxy_pass $auth;
        	}
		location /bar {
			set $auth http://authentication.ecs-demo.test:8083;
			proxy_pass $auth;
        	}
                ...
	}	
```

Accepted Answer

Hello.

**Networking and Firewalls:**
Ensure that the network policies/firewalls in your ECS setup allow for communication between the NGINX service and the authentication service on the necessary ports (8999 and 8083, based on your config).

**Service Discovery:**
Make sure that authentication.ecs-demo.test is being resolved correctly by NGINX. Sometimes, DNS resolution can be an issue within container orchestration systems. You might want to try and replace it with the internal IP address of the service temporarily to rule out DNS issues.

**Logs and Errors:**
If you're tailing /dev/stdout and /dev/stderr and not seeing logs, it might be possible that NGINX is failing before it writes anything meaningful. Ensure that NGINX is actually able to start successfully and listen on the desired port.

**Proxy Parameters:**
Often, when working with proxying requests, it's necessary to forward original headers and handle redirects and errors smoothly. Consider incorporating the following into your location blocks:

```

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
```

**Health Checks:**
Verify if AWS's ECS health checks are configured appropriately for your services and if the services are being marked as healthy.

Ensure that your task definition has the correct port mappings.
Confirm that your ECS service has the correct task definition and desired task count.
Verify if the security group attached to your ECS service allows for inbound traffic on ports 8999 and 8083.

Regards, Andrii

NGINX proxy on ECS fails with 502 bad Gateway

Relevant content