We have instances running ubuntu 20.04 which use systemd-resolved with a local address of 127.0.0.53
This appears to affect instances that have an ElasticIP assigned to them but not others. I cannot see any other difference.
resolvectl status show that 10.0.0.2 is the dns server in use for both affected and non-affected instances.
On affected instances, using tcpdump I can see that 127.0.0.53 is responding with SERVFAIL and not forwarding anythin to the Amazon DNS service, which is at 10.0.0.2
However if I manually send a request directly to 10.0.0.2 on affected instance:
nslookup x.y.com 10.0.02
it successfully perfroms dns request.
Any ideas?