Working with letsencrypt’s certbot for a Lisp webserver

Every 90 days myletsencrypt certificate expires and I renew it manually. I have to cut and paste data from the certbot script into repl forms to serve the right data on the right path and it’s such a pain that sometimes I put it off until after it expires, and people email me about it, and I feel bad.

The manual process looks something like this (not my actual domain or challenges):

# certbot certonly --manual -d my.example.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for my.example.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Create a file containing just this data:

uaut.gj61B3f7oYQcWZSF4kxS4OFh8KQlsDVtPXrw60Tkj2JLW7RtZaVE0MIWwiEKRlxph7SaLwp1ETFjaGDUKN

And make it available on your web server at this URL:

<a href="https://t.umblr.com/redirect?z=http%3A%2F%2Fmy.example.com%2F.well-known%2Facme-challenge%2FSpZa7sf4QMEFoF7lKh7aT4EZjNWSVcur2jODPQkgExa&t=M2NiYjhlOGQ2YTA0ZWFjY2MzNTdmZTgxNWViMDdjN2FiOGU3MjU3ZCxVYlJuR1AzVg%3D%3D&b=t%3A2_iwByQrOjVinKNIchYBLg&p=https%3A%2F%2Flispblog.xach.com%2Fpost%2F189499356038%2Fworking-with-letsencrypts-certbot-for-a-lisp&m=0">http://my.example.com/.well-known/acme-challenge/SpZa7sf4QMEFoF7lKh7aT4EZjNWSVcur2jODPQkgExa</a>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

There’s no way that letsencrypt’s certbot program will automatically get support for my custom hunchentoot “framework” and I don’t really want to look into adding it myself.

For a while I thought about writing someExpect functionality in SBCL - sb-ext:run-program already supports a :pty option so it wouldn’t be super-difficult. With a theoretical cl-expect you could spawn certbot with the right options, slurp out the verification secret and URL via the process’s standard output stream, and call whatever code you need to serve the secret at that location.

I realized today there’s a halfway solution that takes the worst cutting and pasting out of the loop. The unix commandscript –flush starts an interactive session where all input and output is saved to a file. My Lisp webserver can then read certbot program output from that file and configure the webserver automagically for me. I still have to manually start certbot and enter a few REPL commands, but it’s easier than before.

Here’s the core of Lisp side of things for processing the script file:

(defun scrape-letsencrypt-interaction (file)
  (let (secret path)
    (with-open-file (stream file)
      (labels (...)
        (loop
          (skip-past "Create a file containing")
          (next-line)
          (setf secret (read-trimmed-line))
          (skip-past "make it available")
          (next-line)
          (setf path (substring-after ".well-known" (read-trimmed-line))))))))

This could be done just as well (perhaps withcl-ppcre, but I didn’t want to pull it in as dependency.

Here are the functions from labels :

((next-line ()
   (let ((line (read-line stream nil)))
     (when (null line)
       (unless (and secret path)
     (error "Didn't find a secret and path anywhere in ~S"
            file))
       (return-from scrape-letsencrypt-interaction
         (values secret path)))
     line))
 (skip-past (string)
   (loop
     (let ((line (next-line)))
       (when (search string line)
         (return)))))
 (read-trimmed-line ()
   (string-trim '(#\Return)
                (next-line)))
 (substring-after (string target)
   (let ((pos (search string target)))
     (unless pos
       (error "Could not find ~S in ~S" string target))
     (subseq target pos))))

The goal here is to only look at the last secret and URL in the script output, so the main loop keeps track of what it’s seen so far and next-line returns those values at EOF. The output also has ugly trailing ^M noise so read-trimmed-line takes care of that.

Here’s the whole thing all together:

(defun scrape-letsencrypt-interaction (file)
  (let (secret path)
    (with-open-file (stream file)
      (labels ((next-line ()
                 (let ((line (read-line stream nil)))
                   (when (null line)
                     (unless (and secret path)
                       (error "Didn't find a secret and path anywhere in ~S"
                              file))
                     (return-from scrape-letsencrypt-interaction
                       (values secret path)))
                   line))
               (skip-past (string)
                 (loop
                   (let ((line (next-line)))
                     (when (search string line)
                       (return)))))
               (read-trimmed-line ()
                 (string-trim '(#\Return)
                              (next-line)))
               (substring-after (string target)
                 (let ((pos (search string target)))
                   (unless pos
                     (error "Could not find ~S in ~S" string target))
                   (subseq target pos))))
        (loop
          (skip-past "Create a file containing")
          (next-line)
          (setf secret (read-trimmed-line))
          (skip-past "make it available")
          (next-line)
          (setf path (substring-after ".well-known" (read-trimmed-line))))))))

I don’t mind shaving only half a yak when it feels like useful progress!

Someday I’ll get around to a proper Expect-like thing…