ECR Behind an Nginx Proxy

0

Hi Folks,

Due to network security requirements, we have a need to proxy requests to ECR through the following infrastructure

Client -> NLB -> EC2 (Nginx Proxy) -> VPCE -> ECR

We have the API call working to aws ecr get-login-password and we get the token, but the pipe to the docker login command returns:

**.***.**.** - - [28/Nov/2023:02:38:22 +0000] "CONNECT *************.dkr.ecr.us-west-2.amazonaws.com:443 HTTP/1.1" 400 182 "-" "-"

and from the client:

Error response from daemon: Get "https://*************.dkr.ecr.us-west-2.amazonaws.com/v2/": Bad Request

The nginx conf looks (mostly) like:

http {
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;

    server {
        listen              443 ssl;
        server_name         "docker.dev.example.com";
        location / {
            proxy_pass           "https://*************.dkr.ecr.us-west-2.amazonaws.com";
        }
    }
}

I'm not quite sure where I can go next to see what may be missing. Does ECR have any logs to show what isn't there? This works fine outside the proxy from machines that are able to connect directly, so we have narrowed down the issue to the proxy config.

Thanks for any help you can provide.

CajunD
질문됨 6달 전433회 조회
2개 답변
0
수락된 답변

To close the loop on this. My issue was that I was pointing the docker client at the proxy, as well as the proxying endpoint. This was unnecessary. But there were a few additional things that were needed:

  • To log in and to access the repo, it is necessary to use the proxied domain, not the ECR domain, in the docker login call
  • Once the login works, in order to pull, a proxy_redirect is necessary to write the URLs pointing to S3 for the layers. Ex:

proxy_redirect "https://prod-us-west-2-starport-layer-bucket.s3.us-west-2.amazonaws.com" "https://dkr-layer.example.com";

  • Ensure the Host header is rewritten to the ECR domain in the nginx proxy config

Hope this helps others. Cheers!

CajunD
답변함 2달 전
  • Hey, did this work for helm pull as well? I am running into cert issue when pulling from helm but when using docker pull I am able to pull the image

  • I don't use Helm, but I have been successful proxying other types of traffic. It's important to understand if there are any other calls to domains or IP's via redirects, beyond the initial call. Sometimes, when a system has access to the entire internet, then we don't always notice the additional calls. This was one of the issues in my case, the calling machine can ONLY access that single IP, so everything needed to be managed, including the extra request to new domains. I resolved it by watching network traffic on the machine itself. Good Luck.

0

I understand that you need to proxy requests to ECR. Reviewing the nginx configuration, I see that you are delegating the requests to "xxxxxxx.dkr.ecr.us-west-2.amazonaws.com".

Please note that ECR has two different VPC Endpoints -- com.amazonaws.us-east-1.ecr.api com.amazonaws.us-east-1.ecr.dkr

Ensure that you are passing requests to both of the ECR vpc endpoints. Coming to the error you are facing - "Bad Request" to ECR. Bad request error is usually observed when you pass the VPC Endpoint DNS name to the docker login command as shown in the example below:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin vpce-xxxxxxxxxxxxxxx.dkr.ecr.us-west-2.vpce.amazonaws.com

When setting up vpc endpoints for ECR, if you had enabled "enable a private DNS hostname", then a private Route53 record is created in private host zones, which maps "<aws_account_id>.dkr.ecr.<region>.amazonaws.com" to private IP(s) of ECR VPC Endpoint. Therefore you will be able to use the below command


aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.us-west-2.vpce.amazonaws.com

Unfortunately, we would not have ECR Logs that you can refer.

Considering, we need further information and Network specific details to be able to understand the exact reason for the error. Please feel free to open a support case with AWS Premium support to get further assistance.

AWS
지원 엔지니어
답변함 6달 전
  • Thanks Praneel,

    This isn't a VPC Endpoint issue, as I have both of the endpoints created and tested. I can also confirm that I can log in to the ECR from the proxy EC2 (which would use the VPCE) using:

    aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <ecr_id>.dkr.ecr.us-west-2.vpce.amazonaws.

    I have also added a Host header to the proxy config

    proxy_set_header Host <ecr_id>.dkr.ecr.us-west-2.amazonaws.com;

    Still getting the same result.

    I've read that there are several people who have accomplished this, and have tried as much as I could.

    I'll open a request. Thanks.

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠