Tweaking Python programs

If you’re not a Python programmer and you’re trying to customise some Python code, but your changes do not seem to make a difference, then there might be a simple explanation and fix.

I recently installed mailnag to regularly check my IMAP mail account and notify me when new mail arrives.  Mailnag is really extensible — it can notify you by inotify notifications, a panel indicator, playing sound files, launching scripts, or any combination of those four.  Just what I was looking for.

I installed mailnag and the gnome shell extension the usual way (Ubuntu 18.04):

$ sudo apt-get install mailnag
$ sudo apt-get install gnome-shell-mailnag

Then I configured both and got them working.  Great!  Only one problem though — the sound file I specified in ~/.config/mailnag/mailnag.cfg was being played too loud for my liking and there didn’t seem to be any way to adjust the volume.

I tried normalising the volume levels in the sound file (an .mp3) using normalize-audio , but to no avail.  It would seem as though ID3 tags are being ignored.

I didn’t want to re-encode the audio file because its quality was already pretty marginal, so I thought I’d have a go at tweaking mailnag’s Python code and try adjust the volume from within that.

Editing /usr/lib/python2.7/dist-packages/Mailnag/plugins/soundplugin.py I discovered that the program uses GStreamer 1.0 for playback, and this section looked like it was getting down to business:

ply = Gst.ElementFactory.make("playbin", "player")
ply.set_property("uri", "file://" + os.path.abspath(filename))
pt = _GstPlayThread(ply)
pt.start()

I tried adding the following line (immediately after the first set_property), but it didn’t make a difference:

ply.set_property("volume", 0.5)

Responses to some questions over on on Stack Overflow suggested that it was the right way to go, and should work (as it was working for others), so that prompted me to consider the possibility that the problem wasn’t the code itself, rather something else.

I had rebooted the machine between edits, and seen no change, so I didn’t think I was experiencing a disk caching problem.

Although the program was installed globally for all users to access — so should only exist in one (already known) location — I decided to test to make sure the file I was editing was actually the same one as was being used.

I deliberately introduced a syntax error into soundplugin.py by entering random text on a line, saving it, and then restarting mailnag.  The sound still played, and no errors were reported in /var/log/syslog !  That clearly indicated that the .py file was not being called.

Well, if the only file in the whole project that has anything to do with sound isn’t actually playing the sound, what is?

I searched the filesystem for files with a similar name , and found one.  It was in the same folder as soundplugin.py —  the original file (that I had been editing) — and had the same name, but it had a .pyc extension.  Opening the file revealed that its contents were binary, not text.

I asked the Duck “what are .pyc files” and soon found a helpful Stack Overflow question that explained how, in order to reduce startup times, Python compiles (.py) source code files into (.pyc) bytecode files and saves them .  When the program subsequently runs, it looks for a .pyc file first and, if it finds it, it loads and runs that instead of the .py file.

The .pyc file on my system had an old date stamp that indicated that it wasn’t being regenerated as a result of edits to the source.  In an attempt to force it to be regenerated, I simply deleted it.  I figured that would prevent Python from locating and autoloading the .pyc, Python would be forced to look for and load the .py instead, and the .pyc should be recompiled as a result.

I corrected the deliberate syntax error in the .py, restarted the main program, and — finally — the sound file played at a reduced volume.  Success!

Interestingly, a new .pyc file was not generated.  ps u | grep mailnag showed that the mailnag daemon was running under my username, so I checked the permissions on the /usr/lib/python2.7/dist-packages/Mailnag/plugins/ directory and, sure enough, it was owned by root and was mode 755.  Python couldn’t recreate the soundplugin.pyc because it didn’t have write permission to that directory.

sudo chmod 757 relaxed the permissions, I restarted mailnag, and voila — a new soundplugin.pyc file!

To tidy up after myself, I did a sudo chown root:root soundplugin.pyc to make its ownership the same as all the other files in the directory, then a sudo chmod 755 on the plugins directory itself to restore the mode back to what it was before.

Long story short: You can make changes to .py files, but you need to blow away the old .pyc file — and relax permissions on the enclosing folder — in order to force Python to read the edited .py and regenerate the .pyc.

Python is the first language that I’ve come across that auto-saves compiled bytecode on the fly.  It’s an interesting approach, and I now understand the logic, but it does present a bit of a trap for the non-Python folks out there just trying to make small changes to open source software.  Hopefully this post lets at least one person avoid that trap.