The Ubiquiti Diaries: A Site-to-Site VPN Story

Activating ZeroTier - A Virtual SDN

Software-defined networking matured and WireGuard emerged as a credible lightweight VPN protocol in the latter half of the 2010s. Around that time, some companies like TailScale and ZeroTier developed the concept of cloud-managed mesh VPNs on top of WireGuard - primarily with the aim of making it simple to securely connect systems with each other irrespective of their location. It turned out that such solutions had plenty of use-cases - from connecting people on the go with systems back home to LAN parties for gamers with folks spread around the world. ZeroTier can obviously make a better business case for themselves. While the protocol and client software themselves are open-source, it does need a hosting solution to enable relay connections and / or act as coordinators for direct connections between different clients. While most casual users end up using ZeroTier's global network with the free plan (up to 25 nodes), commercial users can sign up for plans allowing more nodes along with other business-centric features. Enterprises can also obtain a license to self-host the entire infrastructure for additional security.

ZeroTier clients are available for a wide variety of platforms. In order to trial out the solution, I started off by signing up and creating a network with two nodes / members. The gallery below gives a quick overview of the relevant features.

Unlike Tailscale which relies on other identity providers like Google, ZeroTier allows sign-ups for users wishing to become a ZeroTier network administrator. There is an optional added layer of protection in the form of multi-factor authentication delivered by Google Authenticator. Upon logging in, the user is presented with the option to create a new network. Each network is allocated a 128-bit ID (16 hex digits). Administrators can configure the network name and also supply a description. Networks can be private or public, with the former requiring administrators to approve the joining of each and every node.

Each ZeroTier network operates on its own subnet - and users have the freedom to choose something that doesn't conflict with their existing subnets. Auto-assignment is also available. In my case, the USG Pro 4 side was replete with clients using the 192.168.0.0/16 subnets, while the UDM side was using 172.16.0.0/16. I opted to go with 10.242.0.0/16 for the ZeroTier network. After configuring these aspects, the focus shifted to the client machines.

After the installation of the ZeroTier One package on a Windows machine on the USG Pro 4 side and a Rocky Linux machine on the UDM side, each node came up with a 10-digit ID that was first authorized by manually adding it via the management web interface (as shown in the gallery). The ZeroTier One CLI was then used on the nodes to make them join the network using its 128-bit identifier.

While the two nodes were able to communicate via the ZeroTier tunnel, I found that pings were prone to packet loss, with around 30 - 40% of the packets getting dropped. The experience with text-based access like SSH was passable, but video streaming was obviously a no-go. A quick perusal of the documentation indicated that a double NAT would cause drastic drop in the user experience - this made sense, given that the machine behind the UDM was behind such a configuration - the CGNAT followed by the NAT in the UDM itself.

Under these circumstances, I had no option but to install the ZeroTier One packages on the gateways themselves. Given the desire to run a site-to-site VPN, this was always going to be on the cards. The install process was further complicated by the MIPS64-based USG Pro 4 on one end and the AArch64-based UDM on the other.

ZeroTier on the USG Pro 4

ZeroTier maintains official packages for the MIPS64-based Ubiquiti products. Though the suggested steps did not work directly, some helpful pointers in the Github bug report resulted in a successful installation.

After joining the network, it became possible to determine the details of the ZeroTier tunnel - in terms of the peers and the connection method to each of them. In the above screenshot, we see direct connections to all the 'planets' - these are servers maintained by ZeroTier to coordinate the connection between the different members of the network. The 10-digit node ID of each member helps in associating the correct machine with the connection status. A direct connection is the ideal scenario, where the data does not flow through a relay server.

ZeroTier on the UDM

After a number of experiments, I determined that it was better to use a containerized version of ZeroTier on the UDM rather than ZeroTier's official ARM64 package. This installation was a two-step process, starting with the 'on-boot-script' from the community-maintained unifios-utilities, followed by starting the zerotier container using the instructions here.

Steps performed with the USG Pro 4 installation were repeated here with the minor modification to call the executable inside the running container. Cross-referencing the node carrying the RELAY tag in the above screenshot with the members list in the gallery, we find that it is one of the machines on the USG Pro 4's side. On the contrary, the machine behind the UDM (on the last line) gets a direct connection.

Usage Experience

Once the two gateways were set up to communicate via a ZeroTier tunnel, setting up the site-to-site VPN was just a matter of placing the appropriate managed routes (refer to the fifth picture in the gallery). The configured routes are automatically pushed to the members by ZeroTier itself. All nodes in the 172.16.0.0/24 subnet behind the UDM are accessible via 10.242.0.2 (the IP of the ZeroTier interface on the UDM). I had initially configured access to the nodes in the 192.168.0.0/16 subnet via 10.242.0.1 (ZeroTier IP on the USG Pro 4). However, I observed that some reboots ended with the routes configured in the ZeroTier web UI getting allocated to a different interface. That was a puzzling debug - wherein the site-to-site VPN would work between the gateways, but fail when clients on either side were involved. Finally, it turned out to be a ZeroTier bug, albeit one that had already been fixed in a later version (not yet available in either the MIPS64 build or the AArch64 Docker image). In the end, I worked around the issue by creating static routes on the UDM side.

The screenshots with the ZeroTier network information were taken recently after I had the whole configuration stabilized. For more than a month after configuring ZeroTier, I had to put up with periodic disconnects (approximately every 30 minutes), wherein the link would drop on the USG side and not come back up until the tunnel was restarted. I ended up writing a script triggered every couple of minutes to check on tunnel status and scheduling it using the config.gateway.json scheme (as this is not configurable via the web UI).

Restarting the tunnel helped, but got progressively worse. After a month or so, the disconnects were such that the tunnel had to be restarted 8 - 10 times before it would come back up again. Around the same time, I also started evaluating Tailscale for the same purpose. Its IP allotment scheme using IPs designated for CGNAT along with overall opaqueness (compared to ZeroTier) meant that it was a bit more challenging to configure for a site-to-site VPN scenario. However, one aspect that did stand out was that it consistently reported the usage of a relay instead of a direct connection. This got me thinking about the ZeroTier setup - once I had OTT streaming work 'reliably' (save for the periodic disconnects), I had not paid much attention to the relay / direct connection. After delving more into the list of peers reported on either side, it turned out that the connection between the USG Pro 4 and the UDM was indeed getting relayed. A quick update to the firewall entries using the web UI (under WAN_LOCAL) to allow UDP communication over port 9993 helped immediately. After the activation of a direct connection between the USG Pro 4 and the UDM, the disconnects stopped.

The nature of commercial off-the-shelf solutions such as the UniFi lineup is such that most of the community-supplied features (such as the installs of ZeroTier that we have done) are not officially supported. Given the possibility of future firmware updates completely blowing these away, I have opted to add a machine each behind each of the gateways to the ZeroTier network. This should give me some degree of protection in the case of either site remaining unstaffed in such a scenario. It can be seen from the peers list that these nodes connect via relays to the gateways at the other end. Performance might not be great, but it should suffice for accessing the gateways over SSH.

With the site-to-site VPN based on ZeroTier providing enough performance for video streaming without interruptions, the time became ripe to add the finishing touches to the setup. Prior to dealing with those, a few benchmarks are in order.

In the direction that matters - OTT streaming from India to the US - I could see speeds of 20 Mbps+. In the reverse direction, I was limited to around 12 Mbps, likely due to my ancient cable modem that is unable to take advantage of the higher upload speeds that Xfinity / Comcast has started to offer.