Sticky Linux firewall solutions of the past

Sticky Linux firewall solutions of the past

In my previous life as a sysadmin of many machines at a school district without a budget, I had a bunch of dumb hacks. One of them was what I did to deal with the fact that we had no firewall for a significant amount of time. I came up with something I called "flypaper".

I noticed that a bunch of dumb hosts would start scanning IP space sequentially looking for responses to their connection attempts. Since we had a fair amount of IP space, it meant I could actually see them hit the "bottom" of our space a good couple of minutes before they got up to the "top" where I kept all of the really interesting stuff.

Since I had Linux CD tower boxes at all of the schools and thus spread all throughout that IP space, I decided to use them as sensors. First, I wrote this tool which would parse a simple config file for firewall rules. I had grown tired of writing firewall scripts by hand during the ipfwadm days, so when ipchains and then iptables came along and changed everything, I switched to another strategy.

I'd put things like "ALLOW TCP a.b.c.d/nn * * 22" in my config file, and those would be translated into ipfwadm, ipchains, or iptables commands as appropriate. I also allowed other ports through where it made sense. I did something really nasty at the bottom: I added a rule which denied anything which hadn't already been handled, logged it, and copied it to userspace. ipchains had this wacky little switch called "-o" which would copy matching packets to the firewall netlink device.

Then, I wrote another dumb little tool which just sat there and read from that device. It had a list of ports it cared about, a list of IP blocks it would ignore, and a server hostname and address. If it saw a packet arrive for a port on its list and the source wasn't trusted, then it would fire off a "report" to the server.

My server was a third program which received those reports, logged them, and then turned around and rebroadcasted the reports to other nodes. They could then react however they liked. Mostly, they would just add a rule to DROP incoming traffic from the bad source and would go on with life.

Things weren't exactly perfect with this scheme. Even though it had seemingly just replaced ipfwadm, ipchains didn't last too long, either. iptables came along and replaced it. Unfortunately, when that happened, the ipchains "-o" switch didn't have an equivalent. I started thinking about continuing the "detect and report" service in a totally different fashion.

Instead of throwing packets at netlink (which was no longer possible with iptables), I'd use iptables to "twist" the bad incoming connections so that they were now pointed a a single port number on the machine. Then I had something which sat on that port and listened for connections. If a host actually managed to connect (that is, the three-way handshake went through), it was assumed to not be some spoofed packet. Then it would report back ("phone home") just like before.

Fortunately, sanity got a hold of me and I decided that this was the wrong way to go about doing things. I made several changes. First, instead of allowing everything and just watching for a few ports, I started dropping all traffic except for that which I expected to see. I also turned off any sort of "reactive" measures, including the flypaper scheme itself.

Basically, my new philosophy was "don't look interesting". There's just so much IP space out there which is totally dead to the world, so if you don't respond to any queries, maybe you don't even exist. It beats the alternative, which is doing something really strange when probed and getting the attention of curious people who will then try to figure it out manually.

Changing to this policy also eliminated a bunch of false positives. One time, a bunch of my users went to some kind of retreat in the mountains. There, they had some kind of "how to use the Internet" class, or something like that, and it involved connecting back to my server. Someone must have fat-fingered a PuTTY config, since this happened:

Jun 24 09:41:04 serv portwatch[92]: Blocking host: a.b.c.d (port 1)

Nobody had any business connecting to port 1 (tcpmux) on that host, so it reacted and blocked them. Unfortunately, they were behind some kind of connection sharing device, and that made *all of them* unable to access my machine. I found out about that one via a phone call and had to jump in and clean it up. Oops.

I learned some interesting things along the way. For instance, did you know that calling iptables for every item in a firewall config file is a spectacularly bad idea? If you watch what's actually happening, you'll see it read the list, do the append or insert operation, then write it back. It does this over and over and over, taking longer every time. At least, it did back in 2004 when I made this change. I don't feel like finding out if it's still like that right now.

Don't do this. Give the painter something else to do. Instead, run iptables-restore once, dump everything you need into it, then COMMIT and you're done.

After this change, I could commit rules 15 times faster. I think I can safely characterize that as worth the trouble.