Inside IPv4 Next Hop won't reply to a Ping

Question

For the life of me, I'm pulling my hair out to get redundancy working on my VPN from On-Prem to my VPC  
  
1) I am able to ping between my On-Prem LAN (10.1.0.0/16) to the VPC (10.2.0.0/16) consistently  
2) I have dual VPN's set-up between my PA Firewall and AWS following all the guides I've found  
3) I am NOT using BGP and using Static Routes \[currently manually created]  
  
My Firewall has a couple ways to monitor the VPN and change the Static Routes via monitoring Next Hop. I've tried numerous ways to get this to work, an can only conclude that on the Inside IPv4 CIDR that the AWS next-hop endpoint isn't responding to a Ping request from my Firewall  
  
For example: \[As assigned by AWS]  
CIDR - 169.254.112.132/30  
Next Hop (AWS Side) - 169.254.112.133  
On-Prem Tunnel Interface - 169.254.112.134  
  
These are in the same subnet, so ACL's/Security Groups/Routing/Firewall Policies shouldn't impact the ability to ping.  Though I've played with all of these with no resolve.  
  
The bottom line is .133 just won't get a ping response from .134, which is crucial for monitoring the connections.  All the guides out there show that this set-up is a best practice and .134 should respond to a ping for me.  
  
Anyone else having issues with this?  Is there something on the AWS side I need to enable to allow the Inside IP for my Virtual Private Gateway to respond to a ping?  Clearly it is allowing LAN to VPC traffic through, so it is functioning as it's the Next Hop in my routing table for 10.1.0.0  
  
Since I am able to ping from LAN to VPC without issue, my fall-back would to use Next Hop between these segments instead of within the Inside IP's, but this is supposed to work..

Answer

I have a session schedule with the Firewall Support team.  They are going to want to prove that the Next Hop IP address responds to pings.  I have set-up an EC2 instance which is able to ping to my remote LAN without a problem, but it too can't ping the Inside IP address for the tunnel.  Any idea how I can confirm that 169.254.x.x assigned to the VPN will respond to a ping?

Answer

Can someone from AWS confirm whether or not the Inside IP address should be able to be Pinged by the other Inside IP Gateway on the other side of the connection?  
  
This is pretty critical for Peer detection and adjustments to Static/Policy Based Routing automation...

Answer

The Next Hop logic in the FW uses the egress interface to generate the ping (which is a 169.254.x.x network).  I don't expect that the private IP address inside the tunnel would be able to ping an IP address on my LAN without some routing

Answer

PA has a zone policy which discards Link-Local IP's which must be disabled.  
  
Ensure to disable Strict IP Address Check  
  
See:

Answer

Well.  After a month of troubleshooting, I am finally able to get a ping response from the AWS gateway.  Of course it was a local firewall issue.  
  
For anyone else who runs into this, check your Zone Protection profile on Palo Alto.

Answer

I am not sure if anyone can see my post.  I've been talking with myself..   
  
Since I couldn't get Static Route based monitoring to work, I decided to try Dynamic BGP.  I run into the same issue again.  
  
In the AWS troubleshooting document "If the configuration settings are correct, then ping the remote BGP peer IP from your local BGP peer IP to verify the connectivity between BGP peers." per:     
  
I wasn't able to ping 169.254.x.x addresses before, so I don't know how this is supposed to work any better..  
  
Is anyone able to help?  Should I be able to ping 169.254.x.x internal IP's on the Gateways?  So far I haven't been able to, and I wonder if AWS documentation is out of date..

Answer

All my testing can only conclude that the 169.254.x.x Inside IP's assigned by AWS no longer respond to pings.  
  
At one time, they must have been pingable, and at some point they no longer are, as the set-up documentation provided currently when hitting "Download Configuration" which provides guidance on setting up the customer gateway device has a bunch of instructions to use the peer 169.254.x.x IP address for fail over detection. Specifically the documentation for Palo Alto firewalls  
  
Whether this is Static Route Path Monitoring, Policy Based Forwarding, or Tunnel Monitoring, they all rely on the ability to ping the other Inside IP for connectivity detection.  
  
I now know more about Routing, Firewall Config, and AWS VPC than I ever wanted to know.  
  
Two options  
  
1) Re-establish the ability to ping the Internal IP so that the documentation provided works  
2) Update the documentation so not to mislead people and waste countless hours  
  
Hoping someone smarter than me has a recommendation

Inside IPv4 Next Hop won't reply to a Ping

Relevant content