README.org

An interface to communicate with Jupyter kernels in Emacs.

What does this package do?

This package provides an API for communicating with a Jupyter kernel via zmq sockets (http://github.com/dzop/emacs-zmq). It utilizes Emacs’ object implementation, eieio, to define Jupyter client and kernel manager classes that can be sub-classed to provide support for any kind of Jupyter frontend in Emacs. Currently, there is a built-in REPL frontend and org-mode source block frontend.

The base Jupyter client provides default implementations for handling common message replies from a kernel that integrate well with Emacs’ built-in features. Below is a partial list of how kernel messages are integrated with Emacs:

• Sending an inspect request will display the inspect reply in the *Help* buffer and previous inspect requests can be revisited by calling help-go-back or help-go-forward while in the *Help* buffer.
• Making completion requests to a kernel is done through the completion-at-point interface.
• If the kernel asks for input from the user, a prompt is displayed in the minibuffer.
• Search through REPL history using isearch.

Support differences between kernel languages

There are many methods that can be extended to take into account differences between kernel languages. This is achieved by providing a method specializer, jupyter-lang, that can be added to the &context section of a method definition. See cl-generic-generalizers.

For example, the Python kernel sends text/plain data in its inspect replies, but most of the time the documentation requested is written using reStructuredText (rST) markup, the officially supported markup language for documentation strings in Python. In emacs-jupyter all insertion of messages into a buffer is handled by the jupyter-insert method. This method is extended to properly highlight rST when an inspect reply message is being inserted and the message is from a Python kernel.

See the files jupyter-python.el and jupyter-julia.el for how these languages are integrated into the emacs-jupyter framework.

How do I install this package?

The recommended way to install this package is through the built-in package manager in Emacs.

Ensure MELPA is in your package-archives

(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/"))

Ensure the latest versions of MELPA packages are available

M-x package-refresh-contents RET

Install Jupyter

M-x package-install RET jupyter RET

Building a package archive using cask

One way to install this package is to build a package archive using cask (https://github.com/cask/cask) to build a local Emacs package file. To do this, clone the repository, enter its directory, and run the following at the command line:

cask package

This creates a file dist/jupyter-0.6.0.tar containing the package archive. To install it

1. Start your Emacs normally
2. Ensure MELPA is in your package-archives
3. M-x package-initialize
4. M-x package-refresh-contents
5. M-x package-install-file ~/path/to/jupyter/dist/jupyter-0.6.0.tar

Manual installation

For a manual installation you can add the repository directory to your load-path and ensure the following dependencies are installed:

(add-to-list 'load-path "~/path/to/jupyter")
(require 'jupyter)

Building the widget support (EXPERIMENTAL)

There is also support for interacting with Jupyter widgets through an external browser. If a widget is to be displayed, an external browser is opened first to display the widget. In this case, Emacs acts as a relay for passing messages between the kernel and the external browser.

If you would like to try out this limited support, you will need to have node installed on your system to build the necessary javascript. Then you will have to run the following commands from the root project directory:

make widgets

How does this package compare to other similar packages?

There are two popular packages that implement similar functionality to this one

• Interacts with a Jupyter kernel via org-mode source blocks.

• Scimax is an Emacs starterkit for scientists and engineers that extends ob-ipython.

• A Jupyter notebook interface in Emacs.

emacs-jupyter extends the features of ob-ipython by integrating more with org-mode and providing a better REPL interface to the kernel. For example, ob-ipython currently does not provide a function for org-babel-load-in-session. ob-ipython also starts a new process for every request since it relies on calling a Python script to send and receive messages whereas emacs-jupyter directly uses zmq sockets via emacs-zmq for communication and only starts a process on every new client connection. This difference in how messages are passed between Emacs and a kernel is notable when making completion requests. ob-ipython will incur the overhead of starting up a new process and new sockets on every completion request which can potentially be every keystroke if you type slow enough.

The scimax version of ob-ipython offers some useful extensions to the default ob-ipython experience such as custom keybindings when inside an org-mode source block, selective display of mimetypes, and jumping to source block error locations. Many of these features have also been implemented in emacs-jupyter.

ein is more of a full featured solution for a Jupyter notebook interface in Emacs. The goals of emacs-jupyter and ein are different. ein aims to be a frontend to the Jupyter notebook server API (https://github.com/jupyter/jupyter/wiki/Jupyter-Notebook-Server-API) which is an extra layer between the user and a kernel (https://jupyter.readthedocs.io/en/latest/architecture/how_jupyter_ipython_work.html#notebooks). In addition to being notebook client, ein offers many more powerful features for Python kernels. emacs-jupyter, on the other hand, offers an API that implements the Jupyter messaging protocol for communication with a kernel via zmq sockets. The API tries to integrate the interaction between the user and a kernel with built-in Emacs features. The REPL support and org-mode integration are examples of how the API can be used. In the future, it would be nice to add some kind of notebook interface in emacs-jupyter or at least an efficient conversion process between notebook files and org-mode.

Jupyter REPL

To start a new kernel on the localhost and connect a REPL client to it, run the command jupyter-run-repl. Alternatively you can connect to an existing kernel by supplying the kernel’s connection file to jupyter-connect-repl.

The REPL supports most of the rich output that a kernel may send to a client. If the kernel requests a widget to be displayed, a browser is opened that displays the widget. If the kernel sends image data, the image will be displayed in the REPL buffer. If LaTeX is sent, it will be compiled (using org-mode) and displayed.

Rich kernel output

A Jupyter kernel provides many representations of results that may be used by the frontend, in this case Emacs. Luckily, Emacs provides good support for most of the available representations.

The supported mimetypes along with their dependencies are shown below in order of priority if multiple representations are returned. Note, if a dependency is not available in your Emacs, a mimetype with a lower priority will be used to display output.

Inspection

To send an inspect request to the kernel, press M-i when the cursor is at the location of the code you would like to inspect.

Completion

Completion is implemented through the completion-at-point interface. In addition to completing symbols in the REPL buffer, completion also works in buffers associated with a REPL. For org-mode users, there is even completion in the org-mode buffer when editing the contents of a Jupyter source code block.

REPL history

You can navigate through the REPL history using C-n and C-p or M-n and M-p.

You can also search through the history using isearch. To search through history, use the standard isearch keybindings: C-s to search forward through history and C-s C-r to search backward.

Associating other buffers with a REPL

After starting a REPL, it is possible to associate the REPL with other buffers if they pass certain criteria. Currently, the buffer must have the major-mode that corresponds to the REPL’s kernel language. To associate a buffer with a REPL you can run the command jupyter-repl-associate-buffer.

jupyter-repl-associate-buffer will ask you for the REPL you would like to associate with the current-buffer and enable the minor mode jupyter-repl-interaction-mode. This minor mode populates the following keybindings for interacting with the REPL:

jupyter-repl-persistent-mode

A global minor mode that will persist a kernel connection to a buffer about to be displayed if the current buffer is in jupyter-repl-interaction-mode and the buffer being switched to has the same major-mode. This mode is automatically enabled whenever jupyter-run-repl or jupyter-connect-repl is called.

jupyter-repl-maximum-size

Set the maximum number of lines before the REPL buffer is truncated.

Widget support

There is also support for Jupyter widgets integrated into the REPL. If any of the results returned by a kernel have a widget representation, a browser is opened and the widget is displayed in the browser. There is only one browser per client.

This feature is currently considered experimental and has only been tested for simple uses of widgets. See =jupyter-widget-client=.

Integration with org-mode

For users of org-mode, integration with org-babel is provided through the ob-jupyter library. To enable Jupyter support for source code blocks, add jupyter to org-babel-load-languages.

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (julia . t)
   (python . t)
   (jupyter . t))

Note, jupyter should be added as the last element when loading languages since it depends on the values of variables such as org-src-lang-modes and org-babel-tangle-lang-exts. After ob-jupyter has been loaded, new source code blocks with names of the form jupy-LANG will be available. LANG can be any one of the kernel languages found on your system. See jupyter-available-kernelspecs.

Every Jupyter source code block requires that the :session parameter be specified since all interaction with a kernel is through a REPL. For example, to interact with a python kernel you would create a new source block like so

#+BEGIN_SRC jupy-python :session py
x = 'foo'
y = 'bar'
x + ' ' + y
#+END_SRC

By default, source blocks are executed synchronously. To execute a source block asynchronously set the :async parameter to yes:

#+BEGIN_SRC jupy-python :session py :async yes
x = 'foo'
y = 'bar'
x + ' ' + y
#+END_SRC

Since a particular language may have multiple kernels available, the default kernel used will be the first one found by jupyter-available-kernelspecs for the language. To change the kernel, set the :kernel parameter:

#+BEGIN_SRC jupy-python :session py :async yes :kernel python2
x = 'foo'
y = 'bar'
x + ' ' + y
#+END_SRC

Note, the same session name can be used for different values of :kernel since the underlying REPL buffer’s name is based on both :session and :kernel.

Any of the defaults for a language can be changed by setting org-babel-default-header-args:jupy-LANG to an appropriate value. For example to change the defaults for the julia kernel, you can set org-babel-default-header-args:jupy-julia to something like

(setq org-babel-default-header-args:jupy-julia '((:async . "yes")
                                                 (:session . "jl")
                                                 (:kernel . "julia-1.0")))

Rich kernel output

In org-mode a code block returns scalar data (plain text, numbers, lists, tables, …), an image file name, or code from another language. All of this information must be specified in the code block’s header arguments, but all of this information is already provided in the messages passed between a Jupyter kernel and its frontends.

When a kernel provides representations of results other than plain text, those richer representations have priority. For example if the kernel returns LaTeX code, the results are wrapped in a LaTeX source block. Similarly for HTML and markdown. If an image is returned, the image is automatically saved to file and a link to the file will be the result of the code block.

Below are the supported mimetypes ordered by priority

• text/org
• image/svg+xml, image/jpeg, image/png
• text/html
• text/markdown
• text/latex
• text/plain

Since it is possible to determine how a result should be represented in org-mode via its MIME type, only a few header arguments are supported. Currently this is only the :file argument for image results and :results raw for inserting raw latex fragments sent by the kernel.

Image output without the :file header argument

For images sent by the kernel, if no :file parameter is provided to the code block, a file name is automatically generated based on the image data and the image is written to file in org-babel-jupyter-resource-directory. This is great for quickly generating throw-away plots while you are working on your code. Once you are happy with your results you can specify the :file parameter to fix the file name.

org-babel-jupyter-resource-directory

This variable is similar to org-preview-latex-image-directory but solely for any files created when Jupyter code blocks are run, e.g. automatically generated image file names.

Deletion of generated image files

Whenever you run a code block multiple times and replace its results, before the results are replaced, any generated files will be deleted to reduce the clutter in org-babel-jupyter-resource-directory.

Editing the contents of a code block

When editing a Jupyter code block’s contents, i.e. by pressing C-c '= when at a code block, =jupyter-repl-interaction-mode is automatically enabled in the edit buffer and the buffer will be associated with the REPL session of the code block (see jupyter-repl-associate-buffer).

You may also bind the command org-babel-jupyter-scratch-buffer to an appropriate key in org-mode to display a scratch buffer in the code block’s major-mode and connected to the code block’s session.

Connecting to an existing kernel

To connect to an existing kernel, pass the kernel’s connection file as the value of the :session parameter. The name of the file must have a .json suffix for this to work.

Remote kernels

If the connection file is a remote file name, i.e. has a prefix like /host:, the kernel’s ports are assumed to live on host. Before attempting to connect to the kernel, ssh tunnels for the connection are created. So if you had a remote kernel on a host named ec2 whose connection file is /run/user/1000/jupyter/kernel-julia-0.6.json on that host, you would specify the :session as

#+BEGIN_SRC jupyter-julia :session /ec2:/run/user/1000/jupyter/kernel-julia-0.6.json
...
#+END_SRC

Password handling for remote connections

Currently there is no password handling, so if your ssh connection requires a password I suggest you instead use key-based authentication. Or if you are connecting to a server using a pem file add something like

Host ec2
    User <user>
    HostName <host>
    IdentityFile <identity>.pem

to your ~/.ssh/config file.

Standard output, displayed data, and code block results

One significant difference between Jupyter code blocks and regular org-mode code blocks is that the underlying Jupyter kernel can request that the client display extra data in addition to output or the result of a code block. See display_data messages.

To account for this, Jupyter code blocks do not go through the normal org-mode result insertion mechanism (see org-babel-insert-result). The downside of this is that, compared to normal code blocks, only a small subset of the header arguments common to all code blocks are supported. The upside is that all forms of results produced by a kernel can be inserted into the buffer similar to a Jupyter notebook.

The implementation of org-mode code blocks is really meant to handle either capturing the standard output or the result of a code block. When using Jupyter code blocks, if the kernel produces output or asks to display extra information, the results are appended to a :RESULTS: drawer.

jupyter-org-interaction-mode

A minor mode that enables completion and custom keybindings when point is inside a Jupyter code block. This mode is enabled by default in org-mode buffers, but only has an effect when point is inside a Jupyter code block.

Custom keybindings inside Jupyter code blocks

You can define new keybindings that are enabled when point is inside a Jupyter code block by using the function jupyter-org-define-key. These bindings are added to jupyter-org-interaction-mode-map and are only active when jupyter-org-interaction-mode is enabled.

By default the following keybindings from jupyter-repl-interaction-mode are available when jupyter-org-interaction-mode is enabled

API

Naming conventions

Methods that send messages to a kernel are named jupyter-send-<msg-type> where <msg-type> is any message type. The message types are identical to those defined in the Jupyter spec with _ characters replaced by - characters. So to send an execute-request you would call jupyter-send-execute-request.

Similarly, methods that are responsible for handling messages received from a kernel are named jupyter-handle-<msg-type>.

Methods that require a message type as an argument such as jupyter-add-callback should do so by passing a message type keyword such as :execute-request.

Overview

Classes

Lower level classes

Communicating with a kernel

Initializing a connection

For a jupyter-kernel-client to start communicating with a kernel, the following steps are taken:

1. Initialize the connection using jupyter-initialize-connection
2. Start listening on the client’s channels with jupyter-start-channels

If starting a local kernel process both steps are handled by jupyter-start-new-kernel. For remote kernels, you will have to manually supply the connection JSON file to jupyter-initialize-connection and start the kernel channels.

Sending messages

Once a connection is initialized, messages can be sent to the kernel using the jupyter-send-<msg-type> family of methods, where <msg-type> is any valid request message type, see jupyter-message-types. These methods asynchronously send a message to the kernel using a subprocess associated with each client, see help:jupyter-channel-ioloop, and they each return a jupyter-request object which encapsulates the information necessary for handling reply messages received from the kernel in response to the request.

Receiving messages

There are two ways to handle the reply messages sent by the kernel: (1) subclass the jupyter-kernel-client and override the jupyter-handle-<msg-type> family of methods or (2) attach callbacks to the jupyter-request objects returned by the jupyter-send-<msg-type> methods. Both ways can occur in parallel.

When a message is received, jupyter-handle-message is called on the client to kick off the message handling process. Any callbacks associated with the jupyter-request of the message are evaluated and the appropriate jupyter-handle-<msg-type> method called.

Note, the default handler methods of jupyter-kernel-client are no-ops with the exception of jupyter-handle-input-request which requests input from the user and sends it to the kernel.

jupyter-kernel-client

Represents a client connected to a Jupyter kernel.

Initializing a connection

jupyter-initialize-connection takes a client and a connection file as arguments and configures the client to communicate with the kernel whose connection information is contained in the connection file.

After initializing a connection, to begin communicating with a kernel call jupyter-start-channels.

(let ((client (jupyter-kernel-client)))
  (jupyter-initialize-connection client "kernel1234.json")
  (jupyter-start-channels client))

jupyter-initialize-connection is mainly useful when initializing a remote connection or connecting to an existing kernel. In order to start a new kernel on the localhost use jupyter-start-new-kernel

(cl-destructuring-bind (manager client)
    (jupyter-start-new-kernel "python")
  BODY)

The above code starts a new python kernel and returns the jupyter-kernel-manager object used to manage the lifetime of the local kernel process and the jupyter-kernel-client connected to the manager’s kernel. jupyter-start-channels will already have been called on the returned client when jupyter-start-new-kernel returns.

To create multiple client’s connected to the kernel of a jupyter-kernel-manager use jupyter-make-client.

Starting/stopping channels

To start a client’s channels, use jupyter-start-channels. To stop a client’s channels, jupyter-stop-channels. To determine if at least one channel is alive, jupyter-channels-running-p.

You may access each individual channel by accessing its corresponding slot in a jupyter-kernel-client. To access the shell channel of a client

(oref client shell-channel)

this will give you the jupyter-channel object of the shell channel. By accessing the channel slots of the client, individual channels may be started or stopped.

Making requests to a kernel

To free up Emacs from having to process messages sent to and received from a kernel, an Emacs subprocess is created for every client. This subprocess is responsible for polling the client’s channels for messages and taking care of message signing, encoding, and decoding. The parent Emacs process is only responsible for supplying the message property lists (the representation used for Jupyter messages in Emacs) when sending a message and will receive the decoded message property list when receiving a message. The exception to this is the heartbeat channel which is implemented using timers in the parent Emacs process.

Note, the message property lists should not be accessed directly. There are helper functions which should be used to access the message fields. See Message property lists.

The lifetime of a request

Sending a request to a kernel is done through one of the jupyter-send-<msg-type> methods of a jupyter-kernel-client. The arguments of the Jupyter message that each method represents are passed as keyword arguments, the keywords all have names according to the Jupyter messaging spec but with _ replaced by -. These methods construct the message property lists based on their arguments and pass the constructed message to the jupyter-send method of a client. The jupyter-send method then returns a new jupyter-request representing the sent message.

(jupyter-send-execute-request client :code "1 + 2") ; Returns a `jupyter-request'

When a request is sent, the message ID of the request is added to the client’s request table which maps message IDs to their corresponding jupyter-request objects.

When a message is received from the kernel the request that generated it is found in the request table by using the jupyter-message-parent-id of the message. The slots of the jupyter-request are updated, any callbacks associated with the jupyter-request are run for the message, and the message is dispatched to the appropriate channel handler method of the client (one of the jupyter-handle-<msg-type> methods).

A request is considered complete and is dropped from the request table once a status: idle message has been received for the request and it is not the most recently made request.

jupyter-generate-request

When one of the send methods are called, a jupyter-request object is instantiated by a call to jupyter-generate-request and the instantiated request is returned by the send method so that the caller can attach their callbacks as described above.

Most likely, subclasses would want to attach extra information to a request. For example, an org-mode client that sends an :execute-request based on the contents of a source code block might want to keep track of the code block’s buffer position so that it can insert the results at the right location when they are ready.

This is the purpose of the jupyter-generate-request method. If a jupyter-request object is not general enough for some purpose, a subclass of jupyter-kernel-client can define a new request object, ensuring that the slots of a jupyter-request are included, and return the new type of request when jupyter-generate-request is called for a message.

For example, below is the definition of the jupyter-org-request type for handling requests made in an org-mode buffer

(cl-defstruct (jupyter-org-request
               (:include jupyter-request))
  result-type
  block-params
  results
  silent
  id-cleared-p
  marker
  async)

And the context specializers used are

(cl-defmethod jupyter-generate-request ((client jupyter-org-client) msg
                                        &context (major-mode org-mode))
  ...) ; Return a `jupyter-org-request'

Notice that the major-mode context allows for jupyter-org-request objects to be used by jupyter-generate-request when the request is generated in org-mode buffers and to use the less specialized jupyter-request in other contexts.

jupyter-drop-request

When a request is completed, i.e. when the kernel sends an idle message for a request, you may want to do some final cleanup of the request. This is the purpose of the jupyter-drop-request method, it gets called when an idle message has been received for a kernel but only when the request is not the most recently sent request.

Handling received messages

The handler methods of a jupyter-kernel-client are called whenever the corresponding message is received from the kernel. They are intended to be overwritten by subclasses and most of the default implementations do nothing with the exception of the :input-reply, :comm-open, and :comm-close messages. The :input-reply handler asks for input from the user through the minibuffer and sends it to the kernel whereas the :comm-open / :comm-close default message handlers store the state of open comms in the client’s comms slot.

The handler methods have the following signature

(cl-defmethod jupyter-handle-<msg-type> ((client jupyter-kernel-client) req arg1 arg2 ...)
  BODY)

req will be the jupyter-request object that generated the message. arg1, arg2, … will be the unwrapped message contents passed to the handler, their number of arguments and their order are dependent on the message type. Alternatively you may work with the full message property list by accessing the jupyter-request-last-message slot of the juptyer-request object.

See message callbacks for another way of handling received messages.

A note on boolean arguments

For message types that have boolean message fields, the symbol in the variable jupyter--false represents a false value so when checking the contents of these arguments it is best to explicitly check for t.

(if (eq arg1 t) ...)

This is because there are some ambiguities between translating JSON values to their Emacs Lisp equivalents, since nil in Emacs is used both as signifying false or nothing whereas JSON has null for nothing.

Client local variables

Some variables which are used internally by jupyter-kernel-client have client local values. For example the variable jupyter-include-other-output tells a jupyter-kernel-client to pass IOPub messages originating from a different client to their corresponding handlers and defaults to nil, i.e. do not handle IOPub messages from other clients. To modify a client local variable you would use jupyter-set

(jupyter-set client 'jupyter-include-other-output t)

and to retrieve the client local value, use jupyter-get

(jupyter-get client 'jupyter-include-other-output)

These functions just set/get the value of a buffer local variable in a private buffer of the client. You may work with these buffer local variables directly by using the jupyter-with-client-buffer macro, just be sure to use setq-local if you are setting a new client local variable otherwise you may change the global value of the variable. Alternatively you can define a variable as automatically buffer local when set with defvar-local.

(jupyter-with-client-buffer client
  (message "jupyter-include-other-output: %s" jupyter-include-other-output)
  (setq-local jupyter-include-other-output (not jupyter-include-other-output)))

Channel hooks

The channel hook variables jupyter-iopub-message-hook, jupyter-shell-message-hook, and jupyter-stdin-message-hook are all client local variables and functions can be added to or removed from them using jupyter-add-hook and jupyter-remove-hook. See Channel hooks.

jupyter-kernel-manager

Manage the lifetime of a kernel on the localhost.

Kernelspecs

To get a list of kernelspecs on your system, as represented in Emacs, use jupyter-available-kernelspecs which processes the output of the shell command

jupyter kernelspec list

to construct the list of kernelspecs. jupyter-available-kernelspecs also supports remote hosts. If the default-directory points to a remote system, the returned kernelspecs are those on the remote system.

To find all kernelspecs whose kernels match some regular expression use jupyter-find-kernelspecs. In case you would like to get the kernelspec for a specific kernel, use jupyter-get-kernelspec.

You may also use jupyter-completing-read-kernelspec in an interactive spec to ask the user to select a kernel from the list of available kernelspecs.

Managing the lifetime of a kernel

Starting a kernel

As was mentioned previously, to start a new kernel on the localhost and create a connected client, use jupyter-start-new-kernel which takes a kernel name and returns a jupyter-kernel-manager which manages the lifetime of the kernel, and a connected jupyter-kernel-client.

(cl-destructuring-bind (manager client)
    (jupyter-start-new-kernel "python")
  BODY)

Instead of supplying an exact kernel name, you may also supply the prefix of one. Then the first available kernel that has the same prefix will be started. See jupyter-find-kernelspecs.

Stopping a kernel

To shutdown a kernel, use jupyter-shutdown-kernel. To check if a kernel is alive, jupyter-kernel-alive-p.

Interrupting a kernel

To interrupt a kernel, use jupyter-interrupt-kernel.

Making clients connected to a kernel

Once you have a kernel manager you can make new jupyter-kernel-client (or a subclass of one) instances using jupyter-make-client.

jupyter-widget-client

This class adds support for interacting with Jupyter widgets using an external browser for the widget display. In order for this to work properly you will need to have simple-httpd and the websocket packages installed, in addition, you will have to build the required javascript files as described in Widget support.

The default implementation of jupyter-widget-client overrides the following methods of a jupyter-kernel-client

(jupyter-handle-comm-close)
(jupyter-handle-comm-open)
(jupyter-handle-comm-msg)

Comm messages in Jupyter are a way to allow for custom messages between the kernel and a client. In the case of Jupyter widgets they are used to sync widget state between the kernel and client.

It would be amazing to add custom Jupyter widgets to Emacs using the built widget library which would work for widgets such as text boxes, buttons, and other simple widgets, but there doesn’t seem to be a way to support more complex widgets in Emacs that require embedded javascript.

The default implementation of jupyter-kernel-client only keeps track of open comms through a client’s comms slot. The jupyter-widget-client subclass adds the functionality to display and interact with widgets through an external browser. This works by relaying the comm messages between the browser and the kernel through a websocket. For this to work, you will also need to have the simple-httpd and websocket Emacs packages available.

This feature is currently experimental, but seems to work well. I was able to interact with an ipyleaflet map without any noticeable delay.

jupyter-repl-client

jupyter-ioloop

jupyter-channel-ioloop

Callbacks and hooks

There are two main ways of evaluating code in response to a received message from the kernel. You can either subclass jupyter-kernel-client and override the handler methods or you can add message callbacks to the jupyter-request objects returned by the send methods. In both cases, when a message of a certain type is received for a request, the appropriate handler method or callback runs. If both methods are used in parallel, the message callbacks will run before the handler methods.

You can also add a hook to one of the jupyter-<channel>-message-hook client local hooks. Where <channel> can be one of iopub, shell, or stdin.

jupyter-request callbacks

To add callbacks to a request, use jupyter-add-callback. jupyter-add-callback accepts a jupyter-request object as its first argument and alternating (message type, callback) pairs as the remaining arguments. The callbacks are registered with the request object to run whenever a message of the appropriate type is received. For example, to do something when a client receives a :kernel-info-reply you would do the following:

(jupyter-add-callback (jupyter-send-kernel-info-request client)
  :kernel-info-reply (lambda (msg)
                       (let ((info (jupyter-message-content msg)))
                         BODY)))

To print out the results of an execute request:

(jupyter-add-callback (jupyter-send-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (message (jupyter-message-data msg :text/plain))))

To add multiple callbacks to a request:

(jupyter-add-callback (jupyter-send-execute-request client :code "1 + 2")
  :execute-result (lambda (msg)
                    (message (jupyter-message-data msg :text/plain)))
  :status (lambda (msg)
            (when (jupyter-message-status-idle-p msg)
              (message "DONE!"))))

There is also the possibility of running the same handler for different message types:

(jupyter-add-callback (jupyter-send-execute-request client :code "1 + 2")
  '(:status :execute-result :execute-reply)
  (lambda (msg)
    (pcase (jupyter-message-type msg)
      (:status ...)
      (:execute-reply ...)
      (:execute-result ...))))

Channel hooks

Hook variables are available for each channel: jupyter-iopub-message-hook, jupyter-stdin-message-hook, and jupyter-shell-message-hook. Unless you want to run a channel hook for every client, use jupyter-add-hook to add a function to one of the channel hooks. jupyter-add-hook only adds to the client local value of the hook variables.

(jupyter-add-hook
 client 'jupyter-iopub-message-hook
 (lambda (msg)
   (when (jupyter-message-status-idle-p msg)
     (message "Kernel idle."))))

To remove a client local hook, use jupyter-remove-hook.

Channel hooks also provide a way of suppressing the handler methods. If any of the channel hooks return a non-nil value, the handler method for that message will be suppressed.

jupyter-inhibit-handlers

In addition to suppressing handler methods using channel hooks, to prevent a client from running its handler methods for a particular request you can let bind jupyter-inhibit-handlers to an appropriate value before the request is made. For example, to prevent a client from running its stream handler for a request you would do the following:

(let ((jupyter-inhibit-handlers '(:stream)))
  (jupyter-send-execute-request client :code "print(\"foo\")\n1 + 2"))

jupyter-inhibit-handlers can be either a list of message types or t, the latter meaning inhibit handlers for all message types. Alternatively you can set the jupyter-request-inhibited-handlers slot of a jupyter-request object. This slot can take the same values as jupyter-inhibit-handlers.

Waiting for messages

All message passing between the kernel and Emacs happens asynchronously. So if a code path in Emacs Lisp is dependent on some message already having been received, e.g. an idle message, there needs to be primitives that will block so there can be can guarantee that certain messages have been received.

The following functions all wait for different conditions to be met on the received messages of a request and return the message that caused the function to stop waiting or nil if no message was received within a timeout period. The default timeout is jupyter-default-timeout seconds.

For example, to wait until an idle message has been received for a request:

(let ((timeout 4))
  (jupyter-wait-until-idle
   (jupyter-send-execute-request
    client :code "import time\ntime.sleep(3)")
   timeout))

To wait until a message of a specific type is received for a request:

(jupyter-wait-until-received :execute-reply
  (jupyter-send-execute-request client :code "[i*10 for i in range(100000)]"))

The most general form of the blocking functions is jupyter-wait-until which takes a message type and a predicate function of a single argument. Whenever a message is received that matches the message type, the message is passed to the function to determine if jupyter-wait-until should return from waiting.

(defun stream-prints-50-p (msg)
  (let ((text (jupyter-message-get msg :text)))
    (cl-loop for line in (split-string text "\n")
             thereis (equal line "50"))))

(let ((timeout 2))
  (jupyter-wait-until
      (jupyter-send-execute-request client :code "[print(i) for i in range(100)]")
      :stream #'stream-prints-50-p
    timeout))

The above code runs stream-prints-50-p for every stream message received from a kernel (here assumed to be a python kernel) for an execute request that prints the numbers 0 to 99 and waits until the kernel has printed the number 50 before returning from the jupyter-wait-until call. If the number 50 is not printed before the two second timeout, jupyter-wait-until returns nil. Otherwise it returns the stream message whose content contains the number 50.

Message property lists

There is really no need to construct or access message property lists directly. The jupyter-send-<msg-type> client methods already handle creating them by calling the jupyter-message-<msg-type> family of functions. Similarly, when a message is received from a kernel the message properties are unwrapped and passed as arguments to the jupyter-handle-<msg-type> client methods. If required, the message property list is available in the jupyter-request-last-message slot of the jupyter-request passed to the jupyter-handle-<msg-type> client methods.

On the other hand, message callbacks pass the message property list directly to the callback. In this case, the following functions can be used to access the fields of the property list:

;; Get the `:content' propery of MSG
(jupyter-message-content msg)
;; Get the message type (one of the keys in `jupyter-message-types')
(jupyter-message-type msg)
;; Get the value of KEY in the MSG contents
(jupyter-message-get msg key)
;; Get the value of the MIMETYPE in MSG's :data property
;; MIMETYPE should be one of `:image/png', `:text/plain', ...
(jupyter-message-data msg mimetype)

Note that access of the message property lists should only occur through the jupyter-message-* functions since the main parts of a message such as the content and header are lazily decoded.

Convenience macros

jupyter-with-message-content gives a way to extract and bind the keys of a jupyter-message-content easily

(jupyter-with-message-content msg (status ename)
  ...) ; status and ename keys of (jupyter-message-content msg) are bound

There is also jupyter-with-message-data which extracts and binds the mimetypes of jupyter-message-data

(jupyter-with-message-data msg ((res text/plain))
  ...) ; res is bound to (jupyter-message-data msg :text/plain)

Modify behavior depending on kernel language

Since Jupyter supports many different programming language kernels, each with varying degrees of support in Emacs there needs to be a general way of modifying the behavior of the client to take this into account.

This is achieved using the &context specializer of cl-defmethod. There are currently two specializers in use, jupyter-lang and jupyter-repl-mode. jupyter-lang is a context specializer that matches when the kernel language of the jupyter-current-client is equal to the specializer’s argument. For example, below is the function that gets called in the REPL buffer when the kernel language is julia for indenting the current line:

(cl-defmethod jupyter-indent-line (&context (jupyter-lang julia))
  (call-interactively #'julia-latexsub-or-indent))

There are many other entry points where methods may be overridden in such a way. Below is the full list of methods that can be overridden in this way

In addition to the jupyter-lang context, there is also the jupyter-repl-mode context which is identical to the derived-mode context but does its check against jupyter-repl-lang-mode if the jupyter-current-client is a jupyter-repl-client. This is useful to modify behavior depending on the major-mode that is used for a particular language. For example for javascript kernels, it used to setup code highlighting when js2-mode is used as the REPL languages major-mode since js2-mode does not use font-lock.

org-mode

jupyter-org-client

A jupyter-org-client is a subclass of jupyter-kernel-client meant to display the results of a Jupyter code block in an org-mode buffer.

jupyter-org-result

The main entry point for extending how results are inserted into the org-mode buffer is the method help:jupyter-org-result, which dispatches on the MIME type of a result returned from a kernel. The MIME type priority is given in jupyter-org-mime-types. jupyter-org-result can return either an org-element object or a string. In the former case, the org-element is transformed into its string representation before insertion into the buffer. In the later case, the string is inserted into the org-mode buffer as is, without any further processing.

There are helper functions for generating org-element objects which have names like jupyter-org-scalar, jupyter-org-export-block, jupyter-org-file-link, etc.

Extending jupyter-org-result

For a kernel language to extend the behavior of how results are inserted, the jupyter-lang method specializer can be used. For example, below is how :text/plain results are modified for Python code blocks

(cl-defmethod jupyter-org-result ((_mime (eql :text/plain))
                                  &context (jupyter-lang python)
                                  &rest _)
  (let ((result (cl-call-next-method)))
    (cond
     ((stringp result)
      (org-babel-python-table-or-string result))
     (t result))))

cl-call-next-method calls down to a less specialized method of jupyter-org-result and if the returned result is still expected to be plain text, calls org-babel-python-table-org-string to convert any results that look like Python arrays into org-mode tables before returning its result.

jupyter-org-define-key

Bind a key that is only available when point is inside a Jupyter code block. When the command bound to the key is evaluated, jupyter-current-client will be bound to the client of the current code block, also the syntax table will be the same as the underlying kernel language’s (see jupyter-org-with-src-block-client).

These keys only have an effect when jupyter-org-interaction-mode is enabled.