Opening the floodgates of spam after MyDoom.F

Opening the floodgates of spam after MyDoom.F

Yesterday, I wrote about setting up a system to stop spam and certain other types of bad mail. It was never intended to stop worms and other evil attachments. I had stayed out of the "content filtering" business and never touched the body of any message. One day, MyDoom.F strolled in and life got interesting.

For reference purposes, this is approximately what I had running before all of this wormy garbage started:

That's been simplified a little, naturally. The point is that all mail arrived at a system running sendmail which then had a little milter helper on board. That milter chatted with my "mailserv" backend which made the go/no-go decision for each message strictly from the metadata: IP address, HELO, FROM and TO. It did not and could not see the body.

When the SMTP filtering appliance landed, they started barking orders at me. I decided to comply and did exactly what was asked of me. This meant tearing out all of the MX entries in our domain and replacing it with just one pointing at this new box. It would then relay all mail to my main mail server.

Now, since all mail had to go through this thing first, that meant my mail delay scheme was useless. If I delayed something, the appliance would just retry and re-deliver. I had to remove it, in other words. After that, the mail flow was from the outside, to their appliance, then straight into my mail server.

That left us with this scenario:

Oh, you expected a network diagram of the new mess? Okay, here.

Their box (probably) filtered nasty virus / worm type mail by inspecting the attachments. In other words, it did exactly what MIMEDefang and Sophie/Sophos had been doing since I set it up the week before.

The catch is that their box didn't do anything about spam by default. Even when it did, it was limited to what was then considered state-of-the-art like DNSBL checks and simple keyword-based matches. I knew it wouldn't have any concept of the "mail delay" scheme (my name for greylisting at the time), and it would be subject to all of the crap which comes from being exposed.

One of their "network engineers" got stuck with the task of entering a bunch of potentially naughty words into the filter, and then had to enter variants where they use l33t sp34k to avoid those exact filters, and so on. It was a three ring circus. I just watched from the sidelines.

Also, some spammers did "direct-to-A" stuff instead of honoring the MX record pointed at the appliance, so that stuff came right on in. I knew this because I was familiar with the problem space, but the boss and his little pal did not. It was interesting to watch mail come straight in from the outside in the configuration they had approved.

In one afternoon, the entire school district's spam load coupled to the reality that was the spring of 2004. They had been living in an artificial bliss and then suddenly the floodgates opened. It was just like that scene from Ghostbusters when they "shut off the grid" and the entire building explodes as the ghosts all break out of containment at the same time.

Users definitely noticed. They started talking to each other about it. I got wind of this when one of my school tech people and friends forwarded a mail from one of his local users. There's one quote in particular I will never forget.

But I never got all those spam messages until they installed the spam filter!

So true. So very true.

With that move, the boss had decided he was going to run mail in that organization, so I just sat back and let him try. This was someone who didn't realize that "RCPT TO" is what makes mail hit your mailbox and not the "To: ..." which may or may not be in the headers. You can imagine how well that worked out.

Fortunately, I didn't have to put up with it for too much longer. The spring of 2004 clicked by and the end drew near. They apparently legitimately forgot to send me a new set of contracts for their next fiscal year. That's okay. I wasn't going to go for them, anyway.

So, at midnight (their time) on July 1, 2004, I logged out of their systems for the last time and pulled the battery out of my pager. I mailed it back to them a few days later. They had decided to go on without me back in February, and that's exactly what they got.

My nine year tenure at that job had ended. In retrospect, I had stayed there far too long. It took this kind of event to finally open my eyes and force me to take control of my own destiny.

The fact that this single box cost more than they paid me in a year had nothing to do with it. Oh no. Not at all. Why would I care about that?

Epilogue: upon writing this, I just checked back on their primary domain name to see what the mail exchanger situation looks like now. It looks like they've switched to psmtp.com, aka Postini, aka... Google. Oh wow, that should be all sorts of fun for them.