Sporadic SSL error from ubuntu ec2 instances only in us-west-1 to eu-west-1

Question

I've been troubleshooting this issue for a bit, I've noticed that when I spin up any ubuntu instance like 18.04 or 20.04  in the **us-west-1** region and if you try to curl an endpoint that is hosted in the **eu-west-1** region, sometimes the request will fail at the TLS, not always but if you repeat the curl call a couple of times it'll fail.     
  
I've tried this in multiple VPC's including the default vpc with nothing add/configured, different AWS accounts and in different regions and this is only an issue when the instance is ubuntu and hosted in the us-west-1 region and hitting an aws hosted service in the eu-west-1 region.   
  
It does not seem to affect amazon linux based instances.   
  
I did see that someone else had similar experiences as me recently  
https://serverfault.com/questions/1066643/ssl-error-from-aws-server-in-us-west-1-to-eu-west-1  
  
Any ideas?  
  
**Steps to reproduce**  
1) Spin up a ubuntu based instance in us-west-1  
2) ssh into ubuntu instance and attempt to curl a webservice hosted in the eu-west-1 region using https  l.e. https://agcod-v2-eu.amazon.com or https://elasticloadbalancing.eu-west-1.amazonaws.com  
3) the curl call will either fail or succeed, if it succeeds repeat the curl several times, it will eventually fail.   
  
```curl -kv https://agcod-v2-eu.amazon.com  
* Rebuilt URL to: https://agcod-v2-eu.amazon.com/  
*   Trying 52.94.219.81...  
* TCP_NODELAY set  
* Connected to agcod-v2-eu.amazon.com (52.94.219.81) port 443 (#0)  
* ALPN, offering h2  
* ALPN, offering http/1.1  
* successfully set certificate verify locations:  
*   CAfile: /etc/ssl/certs/ca-certificates.crt  
  CApath: /etc/ssl/certs  
* TLSv1.3 (OUT), TLS handshake, Client hello (1):  
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to agcod-v2-eu.amazon.com:443  
* stopped the pause stream!  
* Closing connection 0  
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to agcod-v2-eu.amazon.com:443``  
  
Edited by: Berto on Jun 18, 2021 11:36 AM  
  
Edited by: Berto on Jun 18, 2021 11:37 AM  
  
Edited by: Berto on Jun 18, 2021 11:38 AM

Answer

One of my coworkers was able to compare the system settings that were set for amazon Linux and ubuntu   
And we found out that **net.ipv4.ip_default_ttl**  set to be 64 on ubuntu instances while amazon linux was using 255.   
After updating that to be 255, the ssl issues went away.   
  
Amazon Linux   
```        net.ipv4.conf.default.log_martians=0  
        net.ipv4.conf.default.promote_secondaries=1  
        net.ipv4.conf.default.secure_redirects=1  
        net.ipv4.conf.default.send_redirects=1  
        net.ipv4.ip_default_ttl=255  
        net.ipv4.neigh.default.gc_thresh1=0  
        net.ipv4.neigh.default.gc_thresh2=15360  
        net.ipv4.neigh.default.gc_thresh3=16384  
        net.ipv4.tcp_challenge_ack_limit=1000  
        net.ipv4.tcp_limit_output_bytes=262144  
        net.ipv4.tcp_max_orphans=4096  
        net.ipv4.tcp_max_syn_backlog=128  
        net.ipv4.tcp_max_tw_buckets=4096  
        net.ipv4.tcp_mem=10755  14340   21510  
        net.ipv4.tcp_rmem=4096  87380   6291456  
        net.ipv4.tcp_tw_reuse=0  
        net.ipv4.tcp_wmem=4096  20480   4194304  
        net.ipv4.udp_mem=21510  28680   43020  
        net.netfilter.nf_log.2=NONE```

```net.ipv4.conf.default.log_martians = 1  
net.ipv4.conf.default.promote_secondaries = 0  
net.ipv4.conf.default.secure_redirects = 0  
net.ipv4.conf.default.send_redirects = 0  
net.ipv4.ip_default_ttl = 64  
net.ipv4.neigh.default.gc_thresh1 = 128  
net.ipv4.neigh.default.gc_thresh2 = 512  
net.ipv4.neigh.default.gc_thresh3 = 1024  
net.ipv4.tcp_challenge_ack_limit = 999999999  
net.ipv4.tcp_limit_output_bytes = 1048576  
net.ipv4.tcp_max_orphans = 65536  
net.ipv4.tcp_max_syn_backlog = 1024  
net.ipv4.tcp_max_tw_buckets = 65536  
net.ipv4.tcp_mem = 188412	251218	376824  
net.ipv4.tcp_rmem = 4096	131072	6291456  
net.ipv4.tcp_tw_reuse = 2  
net.ipv4.tcp_wmem = 4096	16384	4194304  
net.ipv4.udp_mem = 376827	502436	753654  
net.netfilter.nf_log.2 = nf_log_ipv4```

Sporadic SSL error from ubuntu ec2 instances only in us-west-1 to eu-west-1

Relevant content