README.org

Org Zettelkasten

This package should be considered work in progress. I use it on a daily basis but the API regularly changes in breaking ways.

I still need to write proper setup instructions. For now, feel free to look around for pieces of code that might be useful to you.

Motivation

When managing large collections of org-files (> 1000), many of the default functions / modes (i.e. org-agenda) become very slow.

This package is an attempt at alleviating this problem by introducing a caching layer that is persisted to disk, and a set of command for interacting with / viewing this data.

The code for this cache can be found at l3kn/org-el-cache.

(Planned) features include:

• A command for finding files by their titles
• Commands for renaming files updating all links to them
• Commands for extracting a headline into a separate file (or doing the inverse)
• A hydra with a bunch of useful hotkeys
• A project view for listing files marked as GTD projects
• A graph view (prototype stage)
• A sidebar showing backlinks for the current file (prototype stage)
• A TODO list that can show tasks collected from hundreds of files (not working at the moment)
• Full-text search (not working at the moment)

Inspiration

David Allen - Getting Things Done

Lion Kimbro - How to Make a Complete Map of Every Thought You Think

Niklas Luhman - Zettelkasten

Similar Projects

• deft, Emacs mode for browsing, filtering & editing directories of files
• EFLS/zetteldeft, Zettelkasten system for deft
• jethrokuan/org-roam, Extensions to deft, graph view
• alphapapa/org-graph-view, Link-graph view for org-mode
• alphapapa/org-ql, Query-language & views for org-mode

If you are not working with > 1000 org-mode files or if you have no yet run into any performance problems, one (or a combination of multiple) of the packages listed above might fit your needs better.

Dependencies

• abo-abo/swiper For searching / selecting files
• abo-abo/ace-link For following links
• abo-abo/hydra

Categories

org-zk-categories is used to customize org-zk settings per directory. Entries should be of the form (:path ... :other-settings ...).

This is most useful for configuring the :setup-fn for a file.

Due to the way this list is searched when determining the category of a file, sub-directories should be listed before their parent directory.

Example

Here is the configuration I’m using for my personal setup:

(defun org-zk-website-setup-fn (title)
  (insert "#+SETUPFILE: ~/org/website/setup.org\n")
  (org-zk-default-setup-fn title))

(setq org-zk-categories
      '((:path "~/org/flashcards/" :name "fc")
        (:path "~/org/sandbox/" :name "sandbox")
        (:path "~/org/archives/" :name "archives" :ignore t)
        (:path "~/org/website/pages/" :name "pages" :setup-fn org-zk-website-setup-fn)
        (:path "~/org/website/" :name "website" :setup-fn org-zk-website-setup-fn)
        (:path "~/org/deft/" :name "zk")
        (:path "~/org/paper/" :name "paper")
        (:path "~/org/" :name "main")))

Alternative Implementations

This could be implemented using Directory Variables, however in that case, there is no one way to see the configuration for all categories and each variable that is set has to be declared non-risky.

Components

org-zk-keywords

File-level attributes are stored as #+KEY: value org mode keywords. These should be placed at the start of the file, not containing any newlines before or between keywords.

• (org-zk-keywords-set-or-add (key value))
• (org-zk-keywords-add (key value))
• (org-zk-keywords-delete (key))

The macro org-zk-def-keyword can be used to create commands to set keywords to one of a list of predefined values using ivy-read. When generating the functions name, the keyword is converted to lowercase and “_” are replaced by “-“.

Example

(org-zk-def-keyword
 "GTD_STATE"
 '("active"
   "someday"
   "planning"
   "cancelled"
   "done"))

Generated function: org-zk-set-gtd-state

org-zk-cache

By default, emacs is not fast enough to efficiently search large collections (>1k files) for TODO keywords, tags, dates etc.

To get around this limitation, a cache is introduced.

This cache works by running org-element-process-buffer each time a file is saved or the buffer moves out of focus (e.g. when switching to another window or buffer).

Sub-modules can register on element types to compute data on a file or headline level that can then be used to implement fast views on all the data in the zettelkasten, e.g. for listing open projects, tasks or calendar entries, generating clocking reports and statistics or for use with the integrated spaced repetition system.

A query language is implemented on this cache for building custom views on the data.

org-zk-links

There are three types of links, parent, child and friend.

When adding a link to some target, a link pointing back to the current file is added to the link list of the target file, using the inverse (parent <-> child, friend <-> friend) link type.

Links to other files that are not part of the main text are placed in a list at the beginning of the file, right after the keywords.

When adding links, the current search term can be used to create a new note using ivy-immediate-done, by default bound to C-M-j.

I’ve rebound it to C-d.

(define-key ivy-minibuffer-map (kbd "C-d") 'ivy-immediate-done)

The org-zk-hydra can be used to quickly add links to the current file. If follow-mode is active, after adding a link the target file is opened. This is useful for quickly creating “mind-maps” without entering text or other links to each file.

org-zk-titlecase

When creating a note using the org-zk commands, the title that is entered is automatically converted to title-case.

This only works for English text and not all rules are implemented. Multi-word conjunctions are not supported yet.

Testing

Integration testing is done by providing an input file, a set of operations and an file with the expected output.

The output is written to a third _got file, which can be diffed with the expected output or used to replace the _expected file if the output generated was valid.

git

I’m willing to sacrifice a bit of memory and meaningful commit messages to make sure no information / history is lost.

Files are committed to version control every hour.

Cache

There are a few other packages that provide more advanced / performant queries on org-mode files:

• org-ql
• org-sql

This package implements a cache for files and headlines using a hash table directly in Emacs, no external database is needed.

To avoid recreating this table on each startup, it can be written to disk at regular intervals and when Emacs is closed.

Each file entry has a hash value that is checked against the files hash on startup. With this, updating the cache for my collection of files takes around 5s.

Cache entries are updated when a .org file is saved, created, moved or deleted.

Derived Tasks

Some tasks can be fully derived from the current state of the rest of the system.

Examples:

• Reviewing due flashcards
• Processing the Inbox
• Reading mail

This could be implemented by skipping the detour through .org files on disk and adding derived tasks to the list when opening the task view.

The downside of this approach is that these tasks would not show up in the default org-agenda an keeping track of their time-tracking information would require an additional database.

Instead, derived tasks are implemented by checking for some condition in regular intervals, then adding a headline to a predefined file.

To avoid cluttering this file, if it already includes a headline with the same title that is not marked as “DONE”, no new entry is added.

Derived tasks are stored in a hash-table to simplify changing the predicate of an existing derived task.

Therefore, each derived task should have a unique task title.

Derived tasks can be registered using the (def-org-zk-derived-task title priority tags predicate) macro.

The example below adds a new task “Process Inbox” if there are at least five entries in the inbox.

(def-org-zk-derived-task "Process Inbox" "A" '("gtd")
  (>= (org-zk-inbox-count) 5))

Project View

Files can be tagged as GTD projects using the #+GTD_STATE keyword.

Based on this keyword, a view of all (active) projects can be created and it’s easy to mark a whole project as on_hold or someday to remove it’s tasks from the task view.