Lambda Docker Container Localhost no such host found

Question

I have a lambda authorizer for my api gateway written in go, running on a docker container. In the init() function of the function, I run a sub process called oathkeeper. Oathkeeper is a piece of open source software that helps with authentication access. I start the service on port 4456. However, when it comes time to call the service with the openapi generated sdk, I get the following error:

Error executing request: Get "http://localhost:4456/decisions": dial tcp 127.0.0.1:4456: connect: connection refused

Why is this happening? I've tried other hosts such as host.docker.internal, but that didn't work either. This is the code where the error is occurring:

func getDecisionsRequest(event events.APIGatewayProxyRequest) sdk.ApiApiDecisionsRequest {
	configuration := sdk.NewConfiguration()

// Set the headers needed by oathkeeper
	configuration.AddDefaultHeader("X-Forwarded-Method", event.HTTPMethod)
	configuration.AddDefaultHeader("X-Forwarded-Uri", event.Path)
	configuration.AddDefaultHeader("X-Forwarded-Host", event.Headers["Host"])
	configuration.AddDefaultHeader("X-Forwarded-Proto", "http")
	configuration.AddDefaultHeader("X-Forwarded-For", event.RequestContext.Identity.SourceIP)
	configuration.AddDefaultHeader("Authorization", event.Headers["Authorization"])

configuration.Scheme = "http"
	configuration.Host = "localhost:4456"

apiClient := sdk.NewAPIClient(configuration)
	request := apiClient.ApiApi.Decisions(context.Background())

return request
}

...

func handler(...) ... {
    resp, err := getDecisionsRequest(event).Execute()
    ....
}

Here is my init function that starts the oathkeeper process:

func init() {
	setJwks()

// Spawn Oathkeeper Server
	cmd := exec.Command("oathkeeper", "--serve", "--config", "/etc/config/oathkeeper/config.yml")
	if err := cmd.Start(); err != nil {
		fmt.Printf("Error starting Oathkeeper server: %v
", err)
		return
	}
}

Answer

Hello Spencer,

When you're running a Lambda function in a Docker container and trying to connect to another service (like Oathkeeper) within the same container, using `localhost` should work, but only if the service you're trying to connect to is running and listening on the specified port. The error `connection refused` suggests that nothing is listening on port 4456 when the Lambda function tries to make the connection.

Items to check:

1. **Service Startup**: Make sure that Oathkeeper starts successfully before the Lambda function tries to connect to it. You can add logging to the init process to confirm that the service has started.

2. **Container Networking**: If `localhost` doesn't work, it could be due to how networking is handled in Docker. In some Docker configurations, the services might not be reachable through `localhost`. Instead, they could be available through the container's IP address.

3. **Port Exposing**: Ensure that the port 4456 is exposed and not blocked by any Docker networking configurations or by the host.

4. **Health Check**: Implement a health check that ensures Oathkeeper is fully initialized and ready to accept connections before your Lambda function tries to connect.

5. **Docker Compose**: If you're using Docker Compose, ensure that your service dependencies are correctly defined, so the services start in the right order.

Here's a simple health check to your init function:

```go
func init() {
    // ... existing code ...

// Wait for Oathkeeper to be ready before proceeding
    for i := 0; i < maxRetries; i++ {
        conn, err := net.Dial("tcp", "localhost:4456")
        if err == nil {
            conn.Close()
            break
        }
        fmt.Printf("Waiting for Oathkeeper to be ready: %v
", err)
        time.Sleep(retryInterval)
    }
}
```

Lastly, if `oathkeeper` runs in a separate container, `localhost` won't work. You'll need to use the Docker networking features to connect containers. This often involves using the service names defined in `docker-compose.yml` as the hostnames.

Hope this helps!

Lambda Docker Container Localhost no such host found

Relevant content