Selling Lisp by the Pound

“Paper late!” cried a voice in the crowd,

“Old man dies!” The note he left was signed,

‘Old Father Kiczales’ - it seems he’s drowned!

Selling Lisp by the pound.

The myth of the Lisp “language”

Some people attempting to sell “Lisp” would characterise Lisp by the same things as the average r/programmerhumor reader: copious numbers of parentheses, and nothing else. Before a Lisp game jam, I asked borodust if I could use a COBOL code generator to write a game, and presented a mockup of a “Hello world” program, a la:

(identification-division
 (program-id :hello-world))
(procedure-division
 (main
  (display "Hello, world!")
  (stop run)))

Maybe he was in on the joke, but he said I could use that. This seems to be what we call Lisp. I have done nothing to the syntax or semantics of COBOL, other than to explicate the “structure” of the syntax with parentheses. It appears only Lispers call, say, C, Lua, JavaScript, etc, etc “ALGOL family” - anyone else would consider those languages’ semantics to be too far out. What kind of “Lisp family” do we have? Are these even “dialects”?

There is no common ground between them. We have none. There is no “Lisp family”, there are no “dialects”. All these languages are mutually incomprehensible, and almost all terrible. The latter part cannot be provided without context, so I will provide some. It suffices to say that, by this definition of what “Lisp” is, it’s surely as painful to use as any other “popular” language.

The simplification of the environment

Common Lisp is a programming system, not just a language: Stuff is updateable, inspectable, etc, etc - large parts of Common Lisp are designed to support interaction and introspection. Almost all type, function and variable bindings can be replaced, there is an interactive debugger, and the way in which this is handled is part of the Common Lisp standard. By this standard alone, it seems that Common Lisp has more in common with Smalltalk, Self or Erlang, than with Racket, Clojure, Fennel and so on.

So, saying the environment is unimportant, like the Awesome Lisp Languages list says, is a big fucking lie, and hosting a Lisp system on another environment, which does not provide such features, can’t be seriously considered a net positive! The experience of the programmer is bounded by the worst parts of the system - which is always going to be the “host” environment in such a situation. Thus, Lisp only really provides an advantage when the host environment is also Lisp.

According to the introduction to that list, we learn that “if we translate the languages to a common syntax, the similarities are more apparent”, and the similarities are that they are mere programming languages, not programming systems, and thus they have poor support for interactive programming and debugging, rarely late bind, and so on. In short, they stink. What I mean by a “programming system” is one of the subjects of Richard P Gabriel’s paper The Structure of a Programming Language Revolution, and in short, a programming system is defined in terms of the running environment, not in terms of features in source code, and there are tools to make conventions easy to follow. Both features are directly dependent on an environment; one does not have the same programming system, if the environment is pulled from under its feet.

And there is rarely an engineering advantage to the “two language” design, where we design fast parts of a program in one language, say C, and then the slow parts in another language, say, Python or Lisp. As far as I know, this is basically only seriously considered in Python, where the reference implementation is some magnitudes slower than even SBCL. But it is socially acceptable to FFI out to C or Java for basically anything, and it’s fine to use Lisp either as a scripting language or driver for some other core logic in those languages.

Well, Lisp is a jealous god. You want productivity, and [… if] that means not cluttering up your system with 25 languages, then you won’t do it if you want productivity. You start falling prey to a ‘productivity’ form of Amdahl’s Law: If you are programming in Lisp and another language (C++, say), and about 50% of the system is written in each, then even if Lisp has infinite productivity, you’ve only speeded things up by a factor of 2. Thus, even a little bit of a non-Lisp language can really throw a monkey wrench into the system, given that Lisp is so much more productive than non-Lisp.

– Henry Baker https://groups.google.com/g/comp.lang.functional/c/xLNt2BU34SA/m/UJdY51y7SZYJ

Taking it all

When a problem appears, it should not be necessary to solve the problem by making other problems harder. One nice thing to model is “identity” of data, i.e. that two structures with the same values are different somehow. Object-oriented languages (and procedural languages with some effort) provide intrinsic “identity” by using references to structures; the memory address is used to form identity. Some functional programmers, particuarly Clojure programmers, said the equivalent of “fuck it, we don’t need identity, cause that doesn’t model time. With pure functions, one can model time.” The book “The Joy of Clojure” calls it a “flip book” model, where one flips through multiple values; but there is no flip book, only pictures and stepping that one has to glue themselves. The book is provided by the user, for better or worse. (Definitely for the worse.)

So how do we achieve both identity and time? Worlds provide STM-like control over side effects http://www.vpri.org/pdf/tr2011001_final_worlds.pdf; they’re a better flipbook, as you have defined pages (worlds), and they give you object identity too. The user only has to provide an ordering of worlds, binding the metaphorical flipbook. Such a design is easily achieved using the meta-object protocol (gasp!) Common Lisp provides, and I did it long ago for kicks. So when I hear that object-oriented languages don’t model “time”, I am more than aware that the exact opposite is true. If someone made a Lisp with a “flip book” time model, and a Lisp with a worlds model, I would be more likely to use the latter, because it does not make problems with object identity harder.

Optimisations

Pareto principle your code: if nice-looking fast code isn’t possible, then localise the ugly fast code, and keep an acceptable model for the rest. It should never be an aim to design a language that makes everything ugly and fast, and such a language should never be used by anyone.

But I lied about “ugly and fast” - they won’t be fast, because you’re up shit’s creek for debugging, and have no room to test out optimisations. I had cobbled up an “interactive” environment for OpenCL for a simulation project, allowing me to rebuild some GPU code quite quickly, and check what the GPU returns before I post-process it on the CPU. With this environment, I was more than able to change some of the code, then observe what it would generate with smaller batches, and check that it looked correct. Provided with this environment (and spare minds provided by ##symbolics2), a 56× speedup in a mere few days of work was achieved. Such a feat would not be possible without live testing and debugging, and it would not be made easier by using parenthesized-prefix syntax to interact with a much less interactive environment.

Following the previous two points, it follows that optimising code should result in still nice-looking code, and no loss of interaction and other qualities. Robert Strandh wrote a letter (published in #sicl) about ideas for a new Common Lisp “standard”, of which the part on block compilation is interesting:

Block compilation is a way to make the code faster by making assumptions that can’t change in the future. But what if there are techniques for making the code as fast anyway, without any particular new feature (I am working on it). Should we then force people to include block compilation anyway in their systems, even though it is not necessary, just so that others who are not that good with their implementations can continue using inferior technology?

Even in the strange world of Scheme, Abelson and Sussman have reminded us that “programs must be written for people to read, and only incidentally for machines to execute.” And this is not at odds with writing high-performance code. To repeat a theme in the introduction, people who write unnecessarily low-level Lisp code are about as ignorant of compiler improvements in the last few decades as people who complain that Lisp must be a slow and interpreted language.

What’s Lisp then?

What’s in a name? discusses how the terms “object-oriented programming” and “incremental compilation” have been abused. The article is interesting in itself, but the most striking part is Pitman’s statement that “object-oriented programming is best characterized as a philosophy, a methodology, a style” and not exactly a set of features.

In a similar fashion, I have attempted to illustrate a style of Common Lisp, in which the “language” is in fact an interactive programming system, features subsume multiple programming paradigms, occasionally “beating them at their own game”, and provides excellent efficiency for high-level, readable programs. Such a style is why one sticks around with a language; there is not another language with the described design strategies.

And when a language designer attempts to “innovate” Lisp, by imitating the design strategies of a more mundane language like C++ or Rust, and disregarding the strategies that made it successful in its niches, they are surely selling Lisp by the pound. The attempts to “sell” Lisp, by dropping out its qualities in favour for more familiar ones, are as misguided as marketing anchovies by selling cans of tuna and calling them anchovies. It is also unlikely that one who falls for this trick would move onto a “real” Lisp, would one make the distinction for marketing purposes, because such properties don’t exist in those languages.

It is for these reasons that I don’t care for Lisp by itself - I am more interested in optimizing compilers which work their magic while also providing late binding, updatable instances, etc, and environments which support efficient execution of high-level programs; so trying to sell Lisp by the pound, and market it by ripping out the actually interesting techniques, is a terrible, terrible idea.

One final thing I should say: it is absolutely possible to sell Common Lisp by the pound. People write libraries which completely break these properties of the language, say by requiring you put all your behaviour into some event loop function, which you can’t change the body of without having to restarting everything. Again, people write CL libraries which are no more than mere aggregations of C interfaces.

So what is one to do? One is to run. If I am to take what I wrote seriously, then I am sure that I am made incredibly powerful with these facilities for program design and optimization. Granted, there are large codebases that are still large even if you can drop the time to program by some two-digit factor, but generally one should seriously consider a reimplementation and perhaps a redesign of any code written without these facilities. It is quasi-common knowledge that the software industry needs a kick up the arse, and taking these features seriously would leave a serious imprint on people who already sold themselves out to inferior programming environments.