Twenty five thousand dollars of funny money

Twenty five thousand dollars of funny money

I used to work at a place that sold ads. One of the things this company wanted was for the employees to try it out and see what it was like to actually use the ads product themselves. It's the usual "dogfooding" thing you hear about sometimes.

To that end, they issued a $250 credit every month. You just had to go to a certain internal web page and click a button, and it would credit it to your account. Every time the calendar rolled over to a new month, you could go click it again.

They told us all about this during our first day or two of classes - the infernally-named "onboarding". I noticed something during this: our presenter hadn't claimed their credit yet, so they went and did it for real right in front of us. They went to load up the page and it bombed - something in the code blew up and it didn't work. They reloaded it and then it worked, and they now had $250 of virtual ad money in their account.

Some weeks later, a new month started and I wanted to get in there and give it a shot. I went to start it up, and it blew up, just like what happened in my class. But hey, this time I had a computer of my own, and access to the source code, and even a tiny bit of experience poking at frontend stuff courtesy of some of the introductory tasks they assigned to new employees. Why not take a whack at it? This place is supposed to be all about fixing random stuff even if it's "not yours" - the "nothing is someone else's problem" posters all over the place implied it, at least.

I loaded it up on my dev environment and got cracking. Sure enough, something was wrong with it, and the first time through, it would blow up. It was something dumb like the code was throwing an exception but the exception handling path was making the wrong sort of log call so that would then blow up the whole request. I fixed the logging so we'd actually get to see what the exception was, and that'd give us a chance to fix any real problems. Simple enough, right? I sent the change to the last person to touch the code... who had just touched it that morning, oddly enough. They thanked me and it was applied.

Then I tried to get my credit, and this time it blew up again, but now it logged what was wrong. I could see this on the dev environment. It was something about calling some function with the wrong number of parameters.

The code itself did something like this:

if (condition) old_func(a, b, c, d, e) else new_func(a, b, c, d, e);

The problem is that new_func didn't take 5 arguments. It took 4. I read through the code and found that it didn't need a "d" argument any more, and so I just changed the arg list to (a, b, c, e). I figured it was a simple oversight by the person who had just changed it.

Then I ran it for myself, clicked the button, got the "your credit is now in your account" message, and was pleased. I asked a friend to try it too and it worked for them as well.

It turned out this very if-then-else part was what had been added that morning, and so I again sent that person the code for review, and they again thanked me and accepted it. I went off to go do other not-frontendy things, and the code went out to the internal web servers a little while later.

A few hours later, someone reached out online: we have to turn off the ads credit thing. It's giving away WAY too much money. How much? Twenty-five thousand dollars. $25,000. Not $250.

What happened? The thing had been passing the credit amount as pennies to "old_func", so it was passing in 25000, because 25000 pennies is in fact 250 dollars. But... new_func took dollars, not pennies. So, 25000 in that context was 25 thousand dollars!

I had been at the company something like six weeks and had changed a line of source code to fix a bug (logging), to uncover another bug (wrong argument count), to enable yet another bug (wrong units, and zero type safety) that gave 25 grand worth of funny money to anyone who clicked! And I had clicked! And I got a friend to click! And other people got it too!

What happened? They just turned off the feature until they could fix it. Those of us who had way too much credit in our accounts turned off our ads so as not to actually consume any of the "bad money", and kept them off until they reversed it out of our accounts. Then we were clear to go back to dogfooding.

And no, nobody was fired for this.

This is yet another reason why I say bare numbers can be poison in a sufficiently complicated system. If that function had demanded a type called "dollars" and the caller had another one called "pennies", it simply would not have passed the type checker/compiler. But, this was before those days, so it sailed right through.