Struggling to connect to RDS through Client-VPN

I’ve set up Client-VPN and can’t seem to reach my RDS instance in a private subnet. My setup looks a little like this:

VPC:

  • CIDR: 10.0.0.0/16
  • DNS Resolution: Enabled
  • DNS Hostnames: Enabled

Client-Vpn:

  • DNS Servers: empty (also tried 10.0.0.2).
  • Security Group: vpn-sg (ingress all from my IP, egress all)
  • Client CIDR: 10.1.0.0/16
  • Transport: UDP 443
  • Associations: 3x private subnets (all have access to RDS instance)
  • Split-tunnel: Enabled

RDS Instance:

  • Security Group: rds-sg
  • Security Group Ingress: All traffic from vpn-sg

Using the AWS VPN Client I can connect to the VPN but then all options for connecting to RDS (telnet, mysql, ping etc) all fail with no route to host. I feel like there is some DNS routing going wrong here and no traffic is going through the VPN but I’m not 100% sure what to do about it.

If it’s relevant, I’m running Ubuntu locally.

Any help would be appreciated!

Edit: If it helps at all (which it may or may not do), once connected to the VPN, I can ping my EC2 instance on a private network IP (10.0.0.x) and once I disconnect, I cannot do that. So I can assume that the VPN is at least working.

Check the Authorizations tab on your Client VPN config and make sure you have authorized ingress to the VPC CIDR for your client VPN endpoint.

“no route to host” suggests that the VPN is not set up correctly. Does your Ubuntu box have a 10.0.0.0/16 IP address after you connect? Any chance that you box has a 10.* address even before you connect with the VPN? You can use tcpdump locally to see what’s going on, or check your VPC flowlog to see if any packets are reaching it from your Ubuntu box.

Is the RDS instance on the same subnet as the EC2 that you can ping? This seems like a case of missing subnet routes.

Also, check (e.g. nslookup hostnameofserver.amazonaws.com) if the hostname actually resolves to an ip in the 10.0.x.x range.

I’d try two things. First setup an ec2 instance on the same subnet as your client vpn endpoint and drop it in the same security group. If you can connect to your RDS instance from there then it’s not a client vpn problem.

Second, are there any more authorization rules? I found out that if you have one group with broad permissions and another group with a subset of those permissions the first group can’t access the second groups resources.

Example. If group A is authorized to 172.0.0.0/16 and group B is authorized for 172.0.0.1/32 then group A cannot access 172.0.0.1.

Do you have a route to the DNS server? There’s a link-local address you can use to avoid ip overlap. You’ll need to add a route to 169.254.169.253/32 and specify that as the client-vpn DNS server.

Did you disable “Publically Accessible” on the RDS

Did you ever figure out the problem? Just curious

I’ve checked and it’s set up correctly. I can access EC2’s by IP so I don’t think there’s any traffic rules missing here.

I’m not sure what you mean? There is a tun0 with 10.1.1.34 as the IP.

My chrome traffic still has the same external IP with or without the VPN.

What am I looking for on the tcpdump?

I’ve added some more details here: amazon web services - AWS Client VPN DNS Resolution for RDS - Stack Overflow

The DNS doesn’t resolve for RDS unless if specifically try dig against the nameserver. However, EC2 does resolve correctly.

I don’t think there’s a missing subnet route here. Traffic can reach RDS on the IP which comes back from the “forced” DNS.

I don’t think it’s a security group issue here. Everything works on IP but not on DNS.

I added more details here: amazon web services - AWS Client VPN DNS Resolution for RDS - Stack Overflow

This is definitely something I don’t understand. Can you give any more information on how to check this?

I had to set my DNS to something useful like Google’s (8.8.8.8, 4.4.4.4) on my studio card and reboot. That was it. I don’t know why but my other DNS wouldn’t resolve.

He might be asking what your home router DHCP scope is. Most are configured for 10.0.0.0/8 or 192.168.0.0/16. If it’s the former, that’ll goof up everything. What is your pc’s IP on your local network? They can’t collide with your VPN scope or your network CIDR. When I set these up I like to choose a VPN DHCP scope of something less usual… 192.168.199.0/24 or something weird.

I believe something is wrong with your DNS client outside of VPN issues. *.*.rds.amazonaws.com should always resolve to the private IP since it’s a public DNS record. The fact that it only resolves when you force your client to use 10.0.0.2 as DNS server is very very odd.

Your dig output shows that you’re using a loopback interface for DNS. By any chance something like pihole blocking domains?

Ah yeah that’s not the case here. There are no overlapping cidrs anywhere.

I don’t have anything like pihole. What’s the loopback interface?

Are you sure that RDS records are public? They would resolve to a private IP which seems like it wouldn’t be in a public DNS record. If you put your RDS instance as publically available, I’d assume they will resolve a private IP inside the VPC and a public one outside of it?

In your stackoverflow post it shows that when you’re trying to resolve the RDS hostname, it uses 127.0.0.53 as a DNS server. Anything in 127.0.0.0/8 is not routable on most networks, so only works on your PC. That’s what’s making me ask if you’re using pihole or any other DNS/security service on your PC.

I’d expect either 192.168.x.x, 172.x.x.x or 10.x.x.x as DNS, or something from your provider or a public DNS server like 8.8.8.8 (actually, try resolving it with a public dns server so dig @8.8.8.8 ***.***.eu-west-1.rds.amazonaws.com)

The DNS in your VPC (10.0.0.2) will only resolve compute.internal and any private hosted zone that’s attached to it. amazonaws.com is a public namespace, but will resolve to private IP’s just fine. It’s not a split horizon DNS domain afaik

This usually means he has some sort of local web protection/antivirus program filtering DNS. He may want to try temporarily disabling that.