How can I use the elastic IP of a NAT Gateway as the source IP over a Site to Site VPN using a Virtual Private Gateway?

As you quoted from the documentation, there's no way to use the Elastic IP over a site-to-site VPN connection: "If you use a public NAT gateway to connect to a transit gateway or virtual private gateway, traffic to the destination will come from the private IP address of the public NAT gateway." Your network is working as documented. There's no way to use AWS-owned public IPs over an AWS site-to-site VPN.

The underlying reason is that both Elastic IPs and automatically assigned public IPv4 addresses are technically implemented as a network address translation (NAT) mechanism at the edge of AWS's network. Packets translated that way can only pass into or out of a VPC via an attached internet gateway (IGW). Packets routed through a VGW never go through that NAT mechanism.

However, VPCs do allow you to use nearly any IPv4 addresses you like in your VPC, including non-AWS public IPs. If you have a small CIDR, perhaps a /27 or at least a /28, of your own public IPs allocated to your on-premises use, which you can spare not to use on premises and can afford to allocate for use in your VPC, you could configure the public CIDR as your VPC's secondary CIDR, configure a subnet with the CIDR or at least a /28 block of it, and place a private NAT gateway in that subnet. If you only send traffic to the VGW after it's passed through the private NAT GW, the packets sent to the VPN would have the IP of the NAT gateway that is internal to the VPC and therefore considered as private by AWS, but which in reality is a public IP allocated to your on-premises use and therefore doesn't overlap any private IP ranges of your customers or partners.

As iBehr correctly advised, if you just need to connect your own VPC to your own on-premises network over a site-to-site VPN, you won't typically need any NATing or public IPs on the AWS side. My advice above is mainly useful in situations where you're a service provider connecting to multiple customers and need a single CIDR on your side that is guaranteed not to overlap the addresses of any customer.