VPC Peering Connection across different region, ping not responding

0

I have an EC2 Instance in ap-south-1 and in eu-west-2 and I am trying to setup a Peering VPC connection to ping from one instance to the other.

Overview

Routing Table for eu-west-2 VPC

DestinationTarget
::/0igw-aaa
0.0.0.0/0igw-aaa
2a05:d01c:273:3700::/56local
172.31.0.0/16local
10.0.16.0/20pcx-ccc

Routing Table for ap-south-1 VPC

DestinationTarget
::/0igw-bbb
0.0.0.0/0igw-bbb
2406:da1a:1d5:f000::/56local
10.0.0.0/16local
172.31.32.0/20pcx-ccc

Security Group in eu-west-2

IP versionTypeProtocolPort rangeSource
IPv4All trafficAllAll10.0.0.0/16

Security Group in ap-south-1

IP versionTypeProtocolPort rangeSource
IPv4All trafficAllAll172.31.32.0/20

I have verified the peering connection is Activated between the two VPCs/subnets. Both regions Network ACLs allow all traffic in and out, similarly the security groups allow all outbound traffic. However, I try and do a simple "ping" to the other instance, no packets are received.

Is there anything wrong with the setup? There is one thing I am unsure about... Is it possible to have an Internet Gateway and a Peering Connection on the same subnet?

Looking at the routing tables, the IP 172.31.40.90 will satisify the 172.31.32.0/20 CIDR block (for the peering connection), and the 0.0.0.0/0 CIDR block for the Internet Gateway... how does it know which route to take if the IP satisfies more than 1 route in the Routing Table?

Update The iptables config on the eu-west-2 was the offending issue.

I reset the iptables using the following commands

sudo iptables -P INPUT ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -F

The offending iptables -L config

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_ACCEPT  all  --  anywhere             anywhere
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_ACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:914
AS0_ACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:915
AS0_ACCEPT  udp  --  anywhere             anywhere             state NEW udp dpt:916
AS0_ACCEPT  udp  --  anywhere             anywhere             state NEW udp dpt:917
AS0_WEBACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_WEBACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:943
AS0_APIACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_APIACCEPT  tcp  --  anywhere             anywhere             state NEW tcp dpt:945

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
AS0_ACCEPT  all  --  anywhere             anywhere             state RELATED,ESTABLISHED
AS0_IN_PRE  all  --  anywhere             anywhere             mark match 0x2000000/0x2000000
AS0_OUT_S2C  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
AS0_OUT_LOCAL  all  --  anywhere             anywhere

Chain AS0_ACCEPT (7 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain AS0_APIACCEPT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain AS0_DNS (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             ip-172-31-0-2.eu-west-2.compute.internal
RETURN     all  --  anywhere             anywhere

Chain AS0_IN (4 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             ip-10-0-0-1.eu-west-2.compute.internal
AS0_IN_POST  all  --  anywhere             anywhere

Chain AS0_IN_NAT (0 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere             MARK or 0x8000000
ACCEPT     all  --  anywhere             anywhere

Chain AS0_IN_POST (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             ip-172-31-32-0.eu-west-2.compute.internal/20
AS0_OUT    all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere

Chain AS0_IN_PRE (2 references)
target     prot opt source               destination
AS0_DNS    tcp  --  anywhere             anywhere             state NEW tcp dpt:domain
AS0_DNS    udp  --  anywhere             anywhere             state NEW udp dpt:domain
AS0_IN     all  --  anywhere             169.254.0.0/16
AS0_IN     all  --  anywhere             ip-192-168-0-0.eu-west-2.compute.internal/16
AS0_IN     all  --  anywhere             ip-172-16-0-0.eu-west-2.compute.internal/12
AS0_IN     all  --  anywhere             ip-10-0-0-0.eu-west-2.compute.internal/8
ACCEPT     all  --  anywhere             anywhere

Chain AS0_IN_ROUTE (0 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere             MARK or 0x4000000
ACCEPT     all  --  anywhere             anywhere

Chain AS0_OUT (2 references)
target     prot opt source               destination
AS0_OUT_POST  all  --  anywhere             anywhere

Chain AS0_OUT_LOCAL (1 references)
target     prot opt source               destination
DROP       icmp --  anywhere             anywhere             icmp redirect
ACCEPT     all  --  anywhere             anywhere

Chain AS0_OUT_POST (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere

Chain AS0_OUT_S2C (1 references)
target     prot opt source               destination
AS0_OUT    all  --  anywhere             anywhere

Chain AS0_WEBACCEPT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
preguntada hace 8 meses646 visualizaciones
3 Respuestas
1
Respuesta aceptada

Ping needs an ICMP rule in the security groups, not an IPv4 rule.

The eu-west-2 security group needs an outbound rule to target 10.0.0.0/16 for ICMP, and also an inbound rule from source 10.0.0.0/16 for ICMP.

And similar for ap-south-1, but using 172.31.32.0/20.

profile picture
EXPERTO
Steve_M
respondido hace 8 meses
  • I've added an "All ICMP - IPv4" rule to Inbound and Outbound SGs in both regions with the correct CIDRs, unfortunately still not working :(

  • Anything on the EC2 instances that could be blocking ping? Such as a host-based firewall like iptables, firewalld or ufw on Linux, or Windows Firewall?

  • +1 to this last comment - check the host firewall.

  • Intrestingly, I can ping the public IP address, but not the private IP address. Why would that be?

  • It was the iptables on the eu-west-2 instance... I shall edit my post and post the offending iptables config... thanks everyone

0

The router will use the more specific rule.

Can you try using the VPC Reachability Analyzer to see if it sheds some light?

profile pictureAWS
EXPERTO
kentrad
respondido hace 8 meses
  • Yes I have used the VPC Reachability Analyzer.. here is the result

    NO_POSSIBLE_DESTINATION: The network component vpc-aaa cannot deliver the packet to any possible destination, or the network component is sending traffic towards a different account or region. See documentation to enable Reachability Analyzer for cross-account analyses.

    Interestingly, if I test it one way it succeeds, if I test it another way it fails... But neither pings work from either side The network is sending traffic to a different region, so I thought this is a limitation of the Reachability Analyzer not being able to analyze across regions.

  • Sorry, missed that limitation.

  • Can you trace from the instance to the peering connection from both sides?

  • Also, I would try opening up the SGs on both side all the way, both incoming and outgoing, to eliminate them as the cause.

  • So, I have opened up the SGs temporarily on both sides, both In and Out-going. So I don't think the SGs or the AGLs are the cause... I think the problem is with the routing table. Could it be because I am confusing Public and Private routing tables and subnets? Do I need a subnet to route to the Internet Gateway, and a separate subnet to route to the Peer Connection or can I use a routing table for both private and public?

0

Hello,

Let me help you with this scenario.

Can you provide me below information:

  • Source IP:
  • Destination IP:
  • Source side of Outbound security group rules, Destination side of Inbound security group rules.
  • Inbound and Outbound NACLs rules.
  • Does both the route table have route pointing out to destination VPC via PCX (VPC peering)?
  • How you are trying to access? Is it IPv4 ICMP request?
  • Have you tried Telnet?
  • Is destination instance windows? Can you verify the OS level firewall rules?
  • Can you launch Linux instance in destination subnet and then try to reach that. What do you see?
  • Can you run a TCPdump on linux box? Do you see incoming packets? #tcpdump -i eth0 src x.x.x.x
  • If you still observing the issue in that case enable flow logs and check for rejected packet.

Thank you.

profile picture
EXPERTO
respondido hace 8 meses
  • Source IP: 172.31.40.90 (private in eu-west-2) Destination IP: 10.0.18.69 (private in ap-south-1) Source Outbound SG: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Destination Inbound SG: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Source ACL: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Destination ACL: Allow all traffic, all protocols, all ports for IPv4 and IPv6 Source Route Table: 10.0.16.0/20 Points to PCX Destination Route Table: 172.31.32.0/20 Points to PCX Yes, ICMP (ping req). No telnet. Both instances Linux. I'll verify firewall

  • Above configuration looks good, can you try these steps as well:

    • Verify route is pointed towards correct peering.
    • Can you run a TCPdump on linux box? Do you see incoming packets? #tcpdump -i eth0 src x.x.x.x. (Try bidirectional)
    • If you still observing the issue in that case enable flow logs and check for rejected packet.

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas