ECR Behind an Nginx Proxy

0

Hi Folks,

Due to network security requirements, we have a need to proxy requests to ECR through the following infrastructure

Client -> NLB -> EC2 (Nginx Proxy) -> VPCE -> ECR

We have the API call working to aws ecr get-login-password and we get the token, but the pipe to the docker login command returns:

**.***.**.** - - [28/Nov/2023:02:38:22 +0000] "CONNECT *************.dkr.ecr.us-west-2.amazonaws.com:443 HTTP/1.1" 400 182 "-" "-"

and from the client:

Error response from daemon: Get "https://*************.dkr.ecr.us-west-2.amazonaws.com/v2/": Bad Request

The nginx conf looks (mostly) like:

http {
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;

    server {
        listen              443 ssl;
        server_name         "docker.dev.example.com";
        location / {
            proxy_pass           "https://*************.dkr.ecr.us-west-2.amazonaws.com";
        }
    }
}

I'm not quite sure where I can go next to see what may be missing. Does ECR have any logs to show what isn't there? This works fine outside the proxy from machines that are able to connect directly, so we have narrowed down the issue to the proxy config.

Thanks for any help you can provide.

CajunD
已提问 6 个月前411 查看次数
2 回答
0
已接受的回答

To close the loop on this. My issue was that I was pointing the docker client at the proxy, as well as the proxying endpoint. This was unnecessary. But there were a few additional things that were needed:

  • To log in and to access the repo, it is necessary to use the proxied domain, not the ECR domain, in the docker login call
  • Once the login works, in order to pull, a proxy_redirect is necessary to write the URLs pointing to S3 for the layers. Ex:

proxy_redirect "https://prod-us-west-2-starport-layer-bucket.s3.us-west-2.amazonaws.com" "https://dkr-layer.example.com";

  • Ensure the Host header is rewritten to the ECR domain in the nginx proxy config

Hope this helps others. Cheers!

CajunD
已回答 2 个月前
  • Hey, did this work for helm pull as well? I am running into cert issue when pulling from helm but when using docker pull I am able to pull the image

0

I understand that you need to proxy requests to ECR. Reviewing the nginx configuration, I see that you are delegating the requests to "xxxxxxx.dkr.ecr.us-west-2.amazonaws.com".

Please note that ECR has two different VPC Endpoints -- com.amazonaws.us-east-1.ecr.api com.amazonaws.us-east-1.ecr.dkr

Ensure that you are passing requests to both of the ECR vpc endpoints. Coming to the error you are facing - "Bad Request" to ECR. Bad request error is usually observed when you pass the VPC Endpoint DNS name to the docker login command as shown in the example below:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin vpce-xxxxxxxxxxxxxxx.dkr.ecr.us-west-2.vpce.amazonaws.com

When setting up vpc endpoints for ECR, if you had enabled "enable a private DNS hostname", then a private Route53 record is created in private host zones, which maps "<aws_account_id>.dkr.ecr.<region>.amazonaws.com" to private IP(s) of ECR VPC Endpoint. Therefore you will be able to use the below command


aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.us-west-2.vpce.amazonaws.com

Unfortunately, we would not have ECR Logs that you can refer.

Considering, we need further information and Network specific details to be able to understand the exact reason for the error. Please feel free to open a support case with AWS Premium support to get further assistance.

AWS
支持工程师
已回答 6 个月前
  • Thanks Praneel,

    This isn't a VPC Endpoint issue, as I have both of the endpoints created and tested. I can also confirm that I can log in to the ECR from the proxy EC2 (which would use the VPCE) using:

    aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <ecr_id>.dkr.ecr.us-west-2.vpce.amazonaws.

    I have also added a Host header to the proxy config

    proxy_set_header Host <ecr_id>.dkr.ecr.us-west-2.amazonaws.com;

    Still getting the same result.

    I've read that there are several people who have accomplished this, and have tried as much as I could.

    I'll open a request. Thanks.

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则