What I mean by "wonky but free"

What I mean by "wonky but free"

Yesterday, John D. Cook picked up on a phrase I used in my Mac rant post from last week: "wonky but free". I used it as a description of how I used to handle things like music and pictures before moving to the Mac, but didn't elaborate.

Today, I will shed some light on some of the situation and show what I used to use for managing my music, and may in fact wind up using yet again not too long from now.

It looks like my music collection started in 1996. MP3s had just started showing up in places where I tended to hang out. I got a few random demos and a couple of real songs, and initially played them in Windows with something called winplay3. I had to play it on my Windows machine since my Linux box was far too slow to handle such things.

About a year later, I put Linux on what had been my Windows machine, and then started using the "l3dec | wavplay" combination to play back MP3s. It was pretty horrible, but it did work. You couldn't really pause, or do anything like back up or go forward, and the pipe meant there was a bit of lag between taking an action and hearing it change, but it did play music. Barely.

At the time, a 5 MB MP3 file was still a big deal, considering that my machines were built around disks in the neighborhood of 200-300 MB. This was around the time that wopr.dorms.tamu.edu was alive, and before it was shut down -- you can guess why. There was a lot of music flying around on our relatively small Internet connections.

Eventually, my stock CDROM drive decided to die, and it was replaced under warranty. Happily, the new one was able to do raw CDDA extraction, and I started the process of "ripping" my music from CD. I no longer needed to download songs which were already right there on little plastic platters.

Now, due to the fact that ripping was slow, encoding was slower, and disk space was at a premium, I tended to only grab the songs which really sounded interesting. This meant the "singles" and a few other ones which I enjoyed from an album. The entire rest of an album went ignored.

This is how things were for a while. I had a directory full of oddly-named MP3 files with no real organization involved. Eventually, I got the bright idea to make this directory visible by my web server so I could just stream out the contents from one system to another. This way, I could talk to people on IRC on one underpowered machine while flinging MP3s around from it, and then I'd stream them over to the other, faster machine, to actually play them.

Of course, streaming was no picnic either. Initially, it happened by clicking on a link in Netscape, and it would then download the entire file and would start its "helper program" to begin playback. This meant I didn't hear anything until it had pulled the entire file across, and that could take several seconds. This gave me an incentive to work on something better.

It was 1998 or 1999 by this point, and I had found something which would play directly from a http URL. The problem then became one of getting that URL to the player. Fortunately, the "m3u" file convention was around, and it was simple enough: drop the URL to your mp3 into it. Then you tell Netscape to hand m3us off to your player as a helper. Downloading a 30 or 40 byte m3u file took no time at all, so the player would start right up and would begin playback as soon as its buffers were ready.

This introduced another problem, though: one of creating all of those .m3u files. I could no longer just click on the automatically-generated directory listings in Apache for the mp3s themselves, since that would do the dreaded "download the whole thing and then play it" thing. I had to make a m3u file for every mp3, and that got old. Even doing it with some scripting magic got annoying, since I had to rerun it any time I renamed something or moved things around. It would also regenerate files which didn't need to be built again, and so on. It was a mess.

I finally wound up making a list and something to parse it. Every entry on that list would turn into a row in a HTML table, and the whole thing would be written to index.html. I just had to write up the list once and run the generator to dump out a fresh music page.

It looked a little like this. Note the old-school way that HTML tables used to be rendered by default with the funky inset borders.

Each album had its own directory, and the "mp3.db" inside of it would be used to generate this HTML output and links to get m3us for all of the songs. This was done by manually changing into the directory and running my generator. On this page, the song's title was a link to the mp3 itself (for downloading purposes, just in case), and that little speaker icon linked to something which would create a m3u for streaming. The actual mp3.db file format was a disaster, but I'll share it here to show how far things have come:

hints/01!Collective Soul!Shine!
hints/02!Collective Soul!Goodnight Good Guy!
hints/03!Collective Soul!Wasting Time!

Yes, I used !s as separators. Wait, it gets better. I used strtok() to split it up. Are you sick yet?

Anyway, these entries refer to files like "hints/01.mp3" relative to the "music root", and those would be turned into URLs like http://(my_host)/mp3/hints/01.mp3".

There was also a top-level generator which read in something called "album.db" and generated links to the individual pages. That file wasn't any better in terms of format:

hints!Collective Soul!Hints Allegations and Things Left Unsaid!

As ugly as this was, it provided a surprising amount of flexibility. I was able to build a list of all of my music and have a program which would pick a random entry from that line. Then it would kick back a fully-formed URL. So, instead of playing a specific song, I could say "play whatever you find", and I'd get something random every time.

The interface changed over the years. At some point, I got on the whole "green and black" kick, most likely from jwz's "Gronk!", and redid my pages to match. I also started scanning the album art (front and back) and the CDs themselves. Some were visible as thumbnails, others showed up in medium form, and there were links to see it full-size.

At this point, the top-level "all albums" page looked a bit like this (and yes, this burns my eyes now, too):

[I've had to crop this a bit. Click to see the rest if you like.]

Other things are noticeable here. I went and did the research to find the release dates of all of my albums, and then added it to the list. The date notation was "YYYY MM DD NN", for the year, month, day, and then sequence number. This way, double albums like "The Fragile" could be forced to always appear in a given order. One of them ("left") is "1999 09 21 01", and the other ("right") is "1999 09 21 02".

By this point, I had changed my generator, too. It would now start in the "music root" and would descend through the directories in search of my metadata information. Upon finding an album, it would generate that album's index page, but it would also add it to the top-level albums list. I had cleaned up my directory structure to have artists with their albums underneath them. Tracks were renamed to nn-the_song_name.xxx, and I had even started using Ogg Vorbis instead of MP3 in some cases.

By 2003 or so, individual album pages looked like this:

[Click for full size.]

Here, everything is visible. The front page album art is right up front and there are links to pictures of the actual disc (since that's frequently interesting) and the back of the CD case. The song titles are still links to the mp3 files, and the ever-present speaker icon links to the m3u generator. Finally, there's a link at the top which plays the entire album in order.

At some point, I climbed out of the dark theme for my pages and redid the colors. I went from dark and gloomy to pastels, and this is the result:

[Yep, you got it. Click to see it full-size.]

Yes, there's a new yellow speaker icon here, and there's much more going on behind the scenes by this point. Clicking play no longer sends you to a m3u to stream on your local machine. Instead, it now sends a command to the music server running on the same machine as the web server. I ran a process called "musicd" which listened to commands over a Unix domain socket.

Clicking on one of the speaker icons would actually kick off a small CGI program. That program's job was to take your request and fling it over the Unix domain socket to the music daemon. Clicking the red speaker meant "drop everything and play this now", while clicking the yellow one just meant "add this to your play list".

The music daemon was smart, too, and it would never just sit there with nothing to do. If it hit the end of a song, it would go into the "biglist" file all by itself and would start playing something random. This would go on forever until you stopped it.

This approach allowed quite a few things to happen through the "web remote". Besides "play" and "queue", there was also "next", "pause", "stop", "volup", "voldn", and my favorite, "block". Picking "block" on a song would add it to my "never play this again" list, and would then immediately find something else to play.

Songs so marked would actually show up as greyed out on the album pages, much like a downvoted post on Hacker News. You could still request it explicitly, but it would never come up in random play.

It was now 2007 or so, and both the Wii and iPhone now existed. I put together variants on these pages which made sense for either a touch type browser or a "arm's length pointing" type browser. You could do the essential parts without too much trouble. I even found ways to read the individual Wii Remote keys, and so up and down controlled volume, left would pause and unpause, right would skip, and "2" would block the song forever. It was pretty good stuff.

The final evolution of this was in the year or so before I switched over to the Mac and iTunes. I ditched the notion of descending into different pages in order to avoid the whole "blinking" thing when a browser switches to another page. Instead, I decided to make it load everything via a single huge index page, and then would just reveal or hide different areas as needed.

When unexpanded, the list looked like this.

Expanding one of the albums inserted the track list and pushed down the rest of the page. You could expand multiple albums at once if so desired.

Clicking on any of the songs would make them play. It was still possible to do the "queue" behavior, though. I made the page so that pressing "p" or "q" would set the behavior of a click. That way, I could just put it in queue mode and leave it up, and anyone who came along and picked something wouldn't interrupt the current song. Their song would play just as soon as it got to that point in the list.

This is all just the playback side. I haven't said anything about how the CDs were ripped and encoded and how all of that metadata came to be populated. Just so everyone's clear, most CDs do not contain any sort of text data about their artist, album, or track names. There's this whole convoluted process you have to do in order to get that through third parties. I'll talk about that some other time.

I hope you enjoyed this trip through cringe-land, and that your own software development history is much nicer and less cruftier than mine.