John Fremlin's blog: manardb: a fast persistent object store for Common Lisp

Waiting for updates: connected

Just before leaving MSI, I documented and released manardb, a fast, persistent object store for Lisp. One of our projects had quite tight performance requirements. We needed to support multi-process simultaneous read and write access to a datastore with a few million objects — all of which were accessed a few times every couple of minutes. We first tried AllegroCache, but the network version had no chance of meeting the performance requirements (we always needed every item in the database, and fetching one object per network transaction was not going to work), and the local file version seemed not to support multiple simultaneous writers.

The next stage was anardb, which has a slightly optimized Lisp s-expression serialisation format, and safe multiprocess transactions. It took us to the next level, but unfortunately with a full size test database we were at least thirty times too slow. Reading in the database was not IO bound, despite the inefficient format, but the huge amount of garbage generated put a lot of pressure on the GC. This would have been the same for an SQL database, if we were to convert the each row of a table into a Lisp object — and if not, we would have had to reconsider how to represent the naturally hierarchical datastructure we were storing.

The third step was to escape from the host Lisp GC and manage memory ourselves. This immediately suggested a memory-mapped binary database, which should be comparable in speed to using normal Lisp objects but with persistence and multi-process access thrown in for free. The meta-object protocol is expressive enough that objects stored in this database can be treated as if they were standard Lisp class instances, and database classes can even inherit from standard Lisp classes.

This early version of manardb is already a capable object datastore. It has nearly the same in-memory performance as a prevalent database, but nearly zero load times, very low pressure on the garbage collector, and the operating system will transparently write it out to its backing filesystem store when the IO system is lightly loaded, and swap it out it the filesystem when the memory system is heavily loaded.

However, there are still many low-hanging fruit where it would be easy to extend its feature-set; they are detailed at the bottom of its homepage. One thing I didn't mention there is supporting free-lists and explicit object deletion — at the moment objects cannot be explicitly destroyed; they must be cleaned up implicitly by one of the two garbage collection algorithms (one does not require Lisp memory space proportional to the datastore size).

Supporting explicit deletion would of course require one to trust the user, and free-lists bring with them complications of their own (the free-list could take up more space than the deleted or existing objects, as each object can be as small as one byte).

I hope that manardb finds applications in other projects. Now that it has API documentation and an official release, and passes an extensive testsuite, I would be interested to hear what the obstacles are to replacing elephant, for example.