Troubleshooting TailScale Network

 3 days ago
source link: https://nyan.im/p/troubleshooting-tailscale-network-en
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is damaged, please click the button below to view the snapshot at that time.

Troubleshooting TailScale Network


May 13, 2022

中文版:一次TailScale网络问题的调试过程 – Frank’s Weblog

As mentioned in an earlier post, I used TailScale to create a mesh network that connects all of my devices, and I used a cloud server located in AliCloud Beijing as an exit node, in order to access geographically restricted network services.

However, I noticed that I could not access the Internet at all when using that exit node. I thought it was a network connectivity issue with the relays, so I didn’t worry too much about it. But afterward, I noticed some other services on that server was not functioning, so I looked into it and found out that the problem was not that simple.

First I noticed that I couldn’t access the internet at all from the server, but curl the IP address was working, which indicated the problem with DNS resolution. resolvectl status showed that there were two DNS servers. I assumed this was the DNS server for the TailScale internal network (actually not, will elaborate later) since the IPs started with 100.100[1],

Link 2 (eth0)
  Current DNS Server:
         DNS Servers:

I tried dig @ baidu.com to check the response from the DNS server and got connection timed out: no servers could be reached. The response from the command became normal after shutting down TailScale. So probably TailScale somehow affected the DNS resolution on the system.


Changing the DNS configuration on the server will work around this problem. Edit /etc/netplan/99-netcfg.yaml, add a public DNS into nameserver section under eth0.

  version: 2
  renderer: networkd
      dhcp4: yes
      dhcp6: no
        addresses: []

Run sudo netplan apply to apply changes, then dig baidu.com returns the correct response.

However, modifying the DNS server allows the server to access the Internet, but many services inside AliCloud still require internal DNS resolution. For example, AliCloud’s internal apt mirror (mirrors.cloud.aliyuncs.com) and products such as cloud databases. Configuring apt sources to public mirrors can be a workaround for the apt mirror issue.

Locating Issue

To locate the problem, we need to find the reason why the IP address is not reachable. I thought these two DNS servers were IPs in TailScale’s internal network, but they were inaccessible by all means. After some searching, I found that and are actually internal DNS servers provided by AliCloud. There are also some AliCloud internal services that use similar IPs, for example, the apt mirror whose IP is, which is also not able to connect using curl.

We can therefore draw a preliminary conclusion that TailScale somehow affected access to the 100.100.x.x IP range.



My first thought was that TailScale was routing the entire 100.100.x.x IP range. However, according to the TailScale documentation, TailScale only routes the assigned IP address, not the entire CIDR. ip route list also confirms this.

ip route list table 52
100.69.x.x dev tailscale0
100.90.x.x dev tailscale0
100.96.x.x dev tailscale0
100.98.x.x dev tailscale0 dev tailscale0
100.104.x.x dev tailscale0
100.121.x.x dev tailscale0
100.127.x.x dev tailscale0

ip route get returns the following result, indicating that the packet will be routed to the eth0 interface. This indicates that the routing table is correct and that the problem is not with the routing. via dev eth0 src uid 0


Another thing that may interfere the packets traveling is iptables. iptables -S reveals the following entries related to TailScale.

-A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xffffffff
-A ts-forward -m mark --mark 0x40000 -j ACCEPT
-A ts-forward -s -o tailscale0 -j DROP
-A ts-forward -o tailscale0 -j ACCEPT
-A ts-input -s -i lo -j ACCEPT
-A ts-input -s ! -i tailscale0 -j RETURN
-A ts-input -s ! -i tailscale0 -j DROP

The last entry of these rules drops the packets to the entire CIDR. The problem was solved after removing the rule using iptables -D.

After some searching, I found there are issues already posted earlier this year:

[1] tailscale drops on firewall when ipv4 is disabled · Issue #3837 · tailscale/tailscale · GitHub

[2] FR: netfilter CGNAT mode when non-Tailscale CGNAT addresses should be allowed · Issue #3104 · tailscale/tailscale · GitHub


To sum up, the problem was caused by a firewall rule set by TailScale to block traffic to CIDR, therefore some services on AliCloud’s internal network were blocked because they reside in this IP range. According to TailScale CLI documentation, adding --netfilter-mod=off parameter when starting TailScale can avoid this rule from being set. However, this poses some security risks.

TailScale set this rule because the IP range ([1] it uses for the TailScale network is reserved for Carrier Grade NAT (CGNAT) and was assumed not to be used by the private networks. However, AliCloud uses this IP range for their internal services, thus causing conflict.


iptables(8) Linux man page

[1] What are these 100.x.y.z addresses? · Tailscale

About Joyk

Aggregate valuable and interesting links.
Joyk means Joy of geeK