The DAT protocol is very fast. Under the hood it’s based on versioned and signed Merkle trees over data streams distributed by a BitTorrent-like P2P network, but there’s also optional support for using HTTP servers as well, meaning you can use a blend of approaches for serving your data: the DAT folks are not religious. Browsing DAT sites feels hardly any different to browsing regular websites, and it supports partial downloads, so you can serve huge datasets. Changes to a site are automatically pushed out over the P2P network, so there’s a natural way to distribute updates to software that’s been wrapped in a DAT site. In fact, the DAT network can propagate updates in real time and Beaker can reload sites when they change … more or less instantly. They even have a proposal for an intents-like system, as I suggested in part two!

Beaker is experimental and they only provide downloads for macOS. Still, what’s there seems designed with good taste. The tools you’d expect and need are all present. It’s a project funded by a non-profit foundation (as opposed to some scammy ICO), so it benefits from full time development. Very often attempts to do a “decentralised web” are stronger on enthusiasm than detail, but Beaker doesn’t seem to have that issue.

Even so. Beaker solves some problems, but app devs are still stuck with the unproductive and injection-prone HTML, CSS and Javascript combo. It’s a start but can we do better?

Back to the UI future

Because Dat sites are versioned, self-signed miniature file systems they’re ideal for distributing code modules of various forms. DAT and Beaker are implemented in node.js, but the next bits I’d pick up for a NewWeb app platform come from the Java world. That’s why in part two I said I’d build on the JVM rather than Node or Electron: running JavaScript on the JVM works better than the other way around and it’d be good to combine DAT with a UI layer that isn’t HTML5.

(note: in case you’re not sure why I want to replace HTML, please read part one)

What UI toolkit could we pick, once we drop HTML? There aren’t many. Qt is good, but very C++ oriented and hard to use from the JVM. We could use each operating system’s native widgets, but then we lose advanced styling — vital for creating the uniquely branded UI themes that people love so much. Java Swing has lots of available components and some great tools, but the design is showing its age, it misses key features and it also isn’t easily styled.

That leaves JavaFX. It’s not well known but it has the features a modern UI toolkit should have like a visual UI designer, rendering using the GPU and cascading style sheets. It supports shader-based effects like blurring, drop shadows, full scene graph animation with timelines, transitions and pluggable interpolators. It supports streaming H.264 video, has a Canvas widget compatible with the HTML5 canvas API and it also gets the boring bits right out of the box: full accessibility, responsive layout managers, HiDPI, charting, bidirectional & complex scripts, fully reactive design, multitouch, printing, date picker, table controls and so on.

Many of the boring bits are basic things HTML should provide but either doesn’t, or has only very recently started to catch up, or can only do with large third party frameworks that are constantly being obsoleted.

It has a few areas where it lags HTML5: in particular it needs a WebGL equivalent. You can do OpenGL in Java but Windows computers often don’t have good GL drivers: Chrome solves this by mapping WebGL to Direct3D behind the scenes.

But we can forgive it for that because a UI built this way is completely invulnerable to XSS! And cutting edge features like menu bars, custom right click menus and keyboard shortcuts actually work. You can also load a few million items into a list view and it won’t kill your machine.

Of course, even though this software has the word “Java” in the name we mustn’t forget that it’s easily accessible from JavaScript, Ruby, Python, Lua, Haskell and so on. Here’s JavaScript that makes a centered Hello World button in a window:

If you’re thinking the code seems a bit odd, don’t worry, you’re just not used to traditional UI toolkits. The HTML equivalent is actually more lines of code:

Plain old JavaFX is good, but still, there’s one framework that would be worth promoting to NewWeb developers: TornadoFX. It provides things that web developers would want like hot reload and MVC, along with other features like easy background task execution, dependency injection, automatic form validation and a type-safe DSL for styling to replace the rather limited CSS.

Distribution, streaming and instant start

One of the reasons the web won is that pushing virtually all app logic onto a server had huge benefits:

• Piracy proof. Do not under-estimate the impact of this.
• Transparent and simple auto update. All users run the same version of your app (approximately: web devs often forget about users who leave tabs open for long periods of time).
• Instant start as the UI is streamed.

It also created problems like high user interface latencies.

Getting this right is tricky. Attempts at app streaming often fail. Java applets got a reputation for loading much slower than web apps. Apps have to be built to stream and the web platform traditionally forced all apps to be written this way, regardless of developer convenience. The trend towards “SPAs” can be summed up as web devs finding ways to toss out the page-by-page app streaming model.

The good news is that bandwidth is much less constrained than it once was. The average web page is now 2mb in size and you can fit a lot of code in 2mb. My NewWeb app browser would understand modules natively. Each module would be a Jigsaw module compressed with pack200. For context, the GSON JSON parsing library takes 37 kilobytes this way, which is smaller than the gzipped CSS file Medium uses to style this blog.

Each module would be a DAT archive. In this way, the root module of an app is versioned with a public key via a P2P network that references yet more archives, either by hash or by key. Widely used modules are widely replicated in the P2P network and can be referenced by hash automatically, driven by build systems and package repositories like Maven or NPM. Attempting to access a class in a module that isn’t downloaded yet either blocks the thread (if it’s a background thread), or re-enters the UI loop to draw a loading animation over the apps UI. All modules are cached. In this way making the app stream intelligently is largely a matter of making sure the modules are small enough.

Sandboxing and permissions

What set of permissions does an app have? The design space for app permissioning is quite wide. Let’s make a first attempt:

• Permissions are associated with the public key that signs the root application module.
• Permissions are always requested via APIs the app browser provides. They are not pre-declared up front. The browser confirms with the user that they wish to grant the requested permission.
• Apps may request multiple storage areas that correspond to individual app-specific file systems. They are, by default, 25 megabytes in size. Each app gets one for free without having to ask permission; this is sufficient to store credentials, tokens, code, cached assets and so on. To get a flexibly sized storage area the app can request the user to name it (this is like saving a file), or it can request a transient storage area of arbitrary larger size which is granted automatically. But these may be deleted by the browser without warning if disk space runs low (this is like automatic uninstallation).
• Apps may open arbitrary sockets on arbitrary ports to arbitrary servers without restriction.

The last one may be surprising. The web doesn’t let apps do this. Nor does Android.

Web apps only have access to “Web Sockets”, which have an HTTP header at the front. This is unfortunate; it effectively prevents web apps from implementing any network protocol agreed in the past 30 years. I find the justification to be rather weak — the idea is to trick proxies into thinking that the connection is a regular web request, and as such WebSockets has various layers of obfuscation to try and ensure that (despite looking very similar) WebSocket servers don’t respond to HTTP and HTTP servers don’t respond to WebSocket requests, nor do caching proxies try to actually cache anything. Proxy piercing is a feature some apps need and others really don’t, probably because they aren’t intended for use in corporate environments or because they are SSL only. We’re already willing to use exotic P2P protocols with HTTP only as a fallback however, so if an individual app wants proxy piercing they can include a WebSockets library and use it that way.

We don’t request user permission to access the network because the only time users ever say no to such requests, what they really want is actually some sort of privacy controls — they aren’t trying to enforce a security policy. Enforcing a concept as vague as ‘privacy’ against a potentially malicious app is something best delegated to the app container rather than the protocol layer, as it may be something that requires constantly shifting implementations and threat models.

The document/app dichotomy

Nearly there!

One of the big benefits of the web is that it supports both documents and apps. Arguably it does neither very well. HTML doesn’t even have a notion of pagination and is awkward to write by hand … but supporting both at once at least reflects the blurry nature of the distinction. This page on MSDN is an app that supports search and theming, and which is also a document, both at the same time.

Traditional desktop apps typically use something like PDF or a stripped down HTML if they want to render a document-like thing, and their own UIs rarely contained significant amounts of styled text. This is because UI toolkits aren’t very good at rendering documents, so there’s a temptation to avoid trying.

I doubt anything clever is needed here. For apps that embed documents a Markdown renderer widget with some markup extensions for styling will get you a long way. That would satisfy both this blog and the MSDN page. The root module can export a service via its module descriptor that yields plain text for search crawlers to easily access.

For documents that embed app UI, the Markdown can let you identify particular areas in which the host app will embed UI panes at render time as they scroll onto the screen.