Why does Cloudwatch show service connect traffic when there is no traffic?

Question

I've enabled Service Connect for a service. The service is also part of a target group, and receives traffic via an Application Load Balancer.

I'm looking at traffic statistics, and have added two metrics to a chart :

* AWS/ApplicationELB app/wwwalb/79fcd6b3b840df2f targetgroup/api-http/b19e3d21366c7997
* AWS/ECS apicluster api-http api

I just enabled Service Connect for the service. Service Connect is configured to accept connections on port 81, while the service runs on port 80 (and receives traffic from the load balancer on port 80).

Surprisingly, the Service Connect statistics show that there are more traffic coming in via Service Connect than via the Application Load Balancer. That puzzles me, as I have not configured any
services to use service connect.

I must be doing something wrong here. What can explain this ECS / Service Connect traffic (when I expected zero traffic)?

Looking at the graph, it even seems that the traffic for ALB and ECS is somehow correlated. However, according to the docs the ECS graph should only graph traffic going via the Service Connect 
proxy. Traffic from the load balancer *should* not pass through the service connect proxy.

Application Load Balancer traffic comes in on port 80 (service port)
Service Connect traffic comes in on port 81 (sidecar proxy port)

![Enter image description here](/media/postImages/original/IMwG5NF8GoRuOuKtEBf-oiOQ)

Answer

Hello sbrattla,

I understand that you want to know why you are seeing traffic related metrics for Service Connect when you do not expect any traffic.

As per AWS ECS documentation, "Amazon ECS configures the task and containers so that applications only connect to the proxy if the application is connecting to the endpoint names in the same namespace. All other traffic doesn't use the proxy. The other traffic includes IP addresses in the same VPC, AWS service endpoints, and external traffic." [1]

If any of the containers expose and listen on a port for network traffic, choose a client-server service. These applications don't need to connect to other client-server services in the same namespace, but the client configuration is configured. A backend, middleware, business tier, or most microservices would use this type of Service Connect configuration. If you want a frontend, reverse proxy, or load balancer application to receive traffic from other services configured with Service Connect in the same namespace, these services should use this type of Service Connect configuration. [2].

[1] Service Connect concepts - Service Connect proxy - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-concepts.html#service-connect-concepts-proxy
[2] Service Connect concepts - Service Connect terminology - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-concepts.html#service-connect-concepts-terms

Why does Cloudwatch show service connect traffic when there is no traffic?

Relevant content