The Ubiquiti Diaries: A Site-to-Site VPN Story
by Ganesh T S on December 21, 2022 8:00 AM EST- Posted in
- Networking
- Ubiquiti Networks
- UniFi
- VPN
The CGNAT Spanner in the Works
Carrier-Grade NAT (CGNAT) is used by ISPs to circumvent the lack of IPv4 addresses available for allocation to their customers. As more and more devices come online and connect to the Internet directly, it is not surprising that there is a dearth of IPv4 addresses to hand out. CGNAT is mostly encountered with WAN connections enabled by mobile service providers (think of smartphones connecting to a 4G or 5G network). However, many wired ISPs (such as Airtel in India) have also started rolling out CGNAT. The easiest way to identify if one is behind a CGNAT is to search for one's IP address on Google. Google helpfully displays your public IP address as the first result. If this address is different from the WAN IP reported by your gateway (and that is usually between 100.64.0.0/16 and 100.127.0.0/16), you are likely behind a CGNAT. This is different from a local IP gathered by a gateway placed behind ISP-supplied equipment that acts as a DHCP server itself.
CGNAT is rarely a problem for regular web browsing and content consumption. Problems start cropping up when dealing with services that require opening up ports on the gateway for bidirectional communication - such as VoIP services, multi-player gaming, and VPN setups. Even simple things taken for granted like systems in the network being able to communicate with a remote NTP server (for time synchronization) are not possible. Since the NAT happens at the ISP end, it is not possible for port forwarding to be configured either. Some services manage to find their way around CGNAT using STUN / TURN, but power users end up finding one roadblock after another. On the whole, CGNAT is a pain in the neck for users accustomed to obtaining public-facing WAN IPs.
NTP Synchronization Attempt behind CGNAT, and the 'Solution' (Using a 'Local' NTP Server)
Ideally speaking, configuring the Site-to-Site Manual IPSec VPN on the USG Pro 4 (having a public WAN IP) with a remote server address of 0.0.0.0, and providing the USG Pro 4's WAN IP as the remote server address on the UDM side (behind the CGNAT) should have worked by allowing the CGNAT side to initiate the site-to-site connection with the server. I did try out this suggestion made by an Ubiquiti employee many years ago - however, the initial key exchange / handshake process appeared to be break down with the response from the USG Pro 4 apparently getting dropped by the UDM's ISP.
An online search for enabling site-to-site VPNs with one site behind a CGNAT brought up two sets of results - one suggesting a connection via an intermediate VPS (cloud server hosted by a 3rd party), and the other trying to make the public WAN IP-possessing side to act as an OpenVPN server and the CGNAT side to act as a client and initiate the connection. I was not interested in bringing a VPS into the picture, and my trials with OpenVPN did not yield any promising results.
At this point, I realized that my Android phone was able to open up a Teleport connection with the UDM despite the gateway going behind CGNAT. So, my first attempt was to get a system to connect to the Teleport VPN and route traffic from specific devices through the system's Teleport VPN connection.
Facebook
Twitter
RSS
35 Comments
View All Comments
prophet001 - Wednesday, December 21, 2022 - link
Not really a ubiquiti fan.Threska - Wednesday, December 21, 2022 - link
Ubiquiti vacuum.OddballSix - Wednesday, December 21, 2022 - link
There's no point in even talking about Ubiquiti, you can't buy most of their products. Some of them have been out of stock in the entire channel for months.Entire parts and lines of products gone. You can't buy them. One breaks? You're screwed. Need to upgrade the firewall? Tough.
HalcyonDays - Wednesday, December 21, 2022 - link
I actually went down a similar path as you did. Years ago, when I moved out, I needed away to troubleshoot my parents network remotely when I inevitably get the dreaded phone call "internet is not working".My requirements for this setup are as follows:
1. Bidirectional encrypted tunnel(s) - preferably peer-to-peer
2. No third-party cloud services
3. Each site access internet through their its own ISP
4. Router at each site will handle the VPN connection - no additional hardware
After attempting and investigating multiple methods, I eventually settled on "tinc" based on the suggestion from the openwrt forums.
"tinc" is a peer-to-peer VPN supported by Tomato, Openwrt, and asuswrt-merlin. It doesn't need all sites to have public IP to work. It just need one site to have public IP (I think). To handle dynamic IPs, I use a free DDNS service and assign a domain name to each of the site.
Since then, I have expanded VPN network to include the in-laws and parents' home in Taiwan. It just required the router at each site to have the public key of at least one other site and it'll be able to see all sites. This means that I can be at any of these sites and still see every site.
Some caveats: I am uncertain of the performance. From what I can tell, "tinc" is pretty lightweight but not as performant as wireguard. Because I don't stream anything over tinc tunnels, I can't vouch for how well it works for for that.
Give it a try.
Samus - Thursday, December 22, 2022 - link
Amazing hardware and stability totally ruined by crap software. The controller is trash. Relying on Java is already a red flag but the way the controller database functions is bazaar and totally insecure. Inheriting\adopting hardware into a new instance results in a mandatory config wipe. No fortune 500 or enterprise network would use this so what really separates it from a $100 consumer product? A consumer product that often has more basic functionality; Ubiquiti has to this day failed to implement MAC cloning, axing it from consideration to anybody who has AT&T or Verizon fiber that need to emulate their gateway from the ONT\media converter. Such a basic feature dating back to the Linksys routers of the 90's missing from a $300 prosumer product is embarrassing and should alone put the company underwater. I mean why?Hamm Burger - Thursday, December 22, 2022 - link
My ISP provides CGNAT by default, but one can pay extra (€1.95/month afair) for a non-fixed but routable address. Which I do. Of course, you have to to know that you can ask, because they don't advertise this feature.Samus - Saturday, December 24, 2022 - link
That is cheap. Commercial block IP's are rarely offered in the US to residential subscribers, and even 'business' internet plans find a way to screw you out of $15 minimum for a "usable" static address.It's worth noting over the years I've seen most IP addresses - even for residential internet - have become statically assigned to subscribers, but they are non-routable.
ballsystemlord - Thursday, December 22, 2022 - link
@Ganesh , why not just contact the ISP and tell them that you were paying for an IP address that is *not* behind CGNAT? I mean, if you're spending the money for the IP you should get it.Jorgp2 - Thursday, December 22, 2022 - link
Yup, or just pay for a /29 or something.coburn_c - Thursday, December 22, 2022 - link
ipv6 is dead and rightly so