Filter all ICMP and watch the world burn

Filter all ICMP and watch the world burn

I helped a friend troubleshoot a problem the other day, and it turned out to be something that first bit me almost 20 years ago. When I talk about the cyclical nature of this industry, I really mean it!

The problem was simple enough: with a certain combination of wireless providers and tunnels, certain web sites became inaccessible. We started from the base report: https://something.or.other/ doesn't work. Trying with 'openssl s_client -connect ...' showed the same thing, so it wasn't the browser. Likewise with curl.

However, doing a curl to the http version of the site worked, but this site just redirects on http requests.

ssh also did not work. It would connect, flip some data back and forth (as shown with -v) and then would hang.

I had another machine with me that had a different setup and used it to jump on the server, at which point I started sshd on another port in debug mode and we kicked off another connection attempt. The same thing happened, and sshd said about what ssh said: it connects, it says some stuff, but gets to the key exchange and hangs.

At some point I decided to leave tcpdump running in the background and then we did another ssh attempt, and somehow, this triggered a memory of my problem from so long ago. Maybe it was the frames being transmitted from the server to the client (and never arriving), but it all clicked into place: "we need to lower the MTU".

I was still on the server and so I ran the 2015 equivalent command to add a host route which has a purposely small MTU. It used "ip", not "route", primarily because this particular connection was over IPv6. It was something like this:

ip -6 route add what:ever/128 via what:ever dev eth0 mtu 960

I picked 960 arbitrarily. It was a good guess, since the next connection attempt worked flawlessly.

Sure enough, when we tried to push some ICMP between the hosts, it disappeared. This is all I needed to see to be sure: in doing this, someone blackholed the very important packets which say "fragmentation needed but DF set". Without them, how can you expect the poor sender to figure it out?

Well, it turns out that in 2015, we have more options. While you wait for the network providers to sort out their ICMP filtering situation, you can enable net.ipv4.tcp_mtu_probing on your server and go on with life. I won't explain how it works here, but in short, it will figure things out even if the magic ICMP responses are missing.

It was gratifying to turn this on (after dropping my route hack) and watch Linux bounce the MTU around until it found the one that worked. It was a little like watching the WOPR crack nuclear launch codes.

Remember how I said this happened to me back in the '90s? It turns out I wrote about it in July 2013. Back then, it was dialup SLIP into BSD/OS. Now it's IPv6 over crazy cellular stuff and crypto tunnels, but the problems remain the same.

Filter ICMP at your peril, folks.

CPE 1704 TKS