Choosing Nim out of a crowded market for systems programming languages - Nim fo...

Choosing Nim out of a crowded market for systems programming languages

I wrote this in a text file more/less to capture my state of mind while moving forward from a pretty rough year, professionally speaking. I hope it might be helpful for anyone who's bewildered with the choices of languages on the market right now, and is in a process of soul searching while considering Nim to pin their personal or company's success upon.

After being jailed up in Ruby, Typescript, and the odd Golang project here and there over the last several years, I've come to the conclusion that I need to settle on a language that's closer to the metal and get rid of the cruft. I started out coding in the mid-90s and refocused my career on Bitcoin and crypto in 2016 after leaving a very large software company in Seattle. C was my first love, but it didn't last long before I was consumed by Java, Oracle, and Solaris.

After the crypto market's meltdown, it's time for me to go back to the drawing board. To get restarted I decided that I need a safe, fast, and statically typed language that can be compiled and targeted to both mobile and desktop environments, as well as being made available to scripting languages via FFI. After struggling with this decision for many years and being on the fence, I feel foolish for not having gone all-in on Nim when I first discovered it, and for having wasted a lot of time pushing higher level application code into the cloud and HSMs when what my industry needs most is to focus on self-custody and self-hosted solutions.

I purchased the excellent Nim in Action book when it first came out and wrote some toy programs. Being a Ruby guy, I stuck with Crystal for my side-projects, and Golang when I needed it, and Ruby for most everything else. I use a lot of C based gems and a lot of proprietary back-end code based on zmq and messagpack. Ruby is a great glue-ware, and I have scaled and studied MRI performance carefully since 2004 but it's a very old crutch, and the promises of a typed and compiled Ruby 3 seem to have been over-promised and under-delivered. A lot of big fintech still uses Ruby, including Coinbase and Stripe. I expect both of them to capitulate to Rust and other statically compiled languages, not that they really need to, but because of the industry optics and need to retain talent.

From the perspective of Nim, here are some broad comparisons to my short-list of candidate languages: Go, Crystal, Swift, and even mruby (whose intermediate .mrb binary is embeddable and callable from C). My runners up: Rust, Zig, and Dart. This isn't a highly scientific post full of esoteric details and language feature matrices. It's about making the best choice for what I can be the most productive in for my target market and product requirements.

Some specific nits to get out of the way:

• mruby is short for eMbedded Ruby, is actually an ISO standard, and unlike MRI code and the Ruby license, is released under MIT. It's also very obscure and used mostly in Japan. I have always reached first for Ruby for back-end code, data migrations, and RPC frameworks. I detest Rails' focus on magic, templates, and its deviation from Ruby's stdlib. I wanted to love mruby, but it effectively means producing a hybrid of ruby intermediate code and C native code. Using mruby gems to bind to C dependencies to work with mruby, just to compile back into C is error prone spaghetti insanity. Pour one out, I tried.
• Crystal is like a Ruby flavored version of Go, and it fits the bill perfectly if the only deployment targets were in the cloud or MacOS/Linux desktop and I only needed to support a higher level API via RPC of some sort. On the surface, Crystal seems like the most direct competitor to Nim in that both have the productivity gains of well known scripting languages with performance characteristics of straight C. In my assessment, I find Nim is much more complete and better optimized.
• Swift is rarely used outside of creating heavy client apps for iOS and macOS. I love the syntax, and protocols click with me better than Rusts's traits for some reason. It was open sourced and has great support for Linux, assuming you use the binaries. Compiling Swift from source is an absolute nightmare and requires huge amounts of ram. It only has one popular package for server-side development, Vapor, which looks like a typical MVC web-app middleware clone; a very high quality and performant clone. Pros for Swift are focus on secure programming outside of just obsessing over memory safety like Rust. But not many people are using Swift for library level development, and it's just not looked upon as a systems level programming language. IBM dropped support for Swift, and Swift's first-tier support in TensorFlow at Google was left hanging in the air after Chris Lattner's departure.
• Go - I will rip Go a bit later in this memo. I love its no nonsense Get Sh!t Done (GSD) factor. But no. Golang is the most reached-for backend languages right now for web programming and stand-alone P2P clients. Try to get closer to the metal and mix mobile into the equation, and it doesn't really work out.

Some definitive Nim wins for my use case are:

1. there's no runtime to carry around after compilation. Go, Crystal, and Swift are all nifty looking languages with fantastic and easy to use syntax, but they incur a big size penalty by compiling a runtime into the executable to support the stdlib. Swift and Crystal are probably the slickest and most fun to code in for my personal tastes, but each are tragically hamstrung as a general purpose language in some way or another, victims of their primary use cases.
2. best support for static linking to existing C code. Because it compiles to C, Nim works seamlessly with C/C++ with no complaints. Go, Swift, and Crystal force you into dynamically linking dependencies which isn't great for packaging low level code. Most blockchain projects are written in Golang and Rust, while Bitcoin core, Lightning reference implementation, Blockstream's Elements, several Bitcoin bip38,39 wallet libs, and Monero's Bulletproofs ZKP implementation are all written in C. The best tested and canonical cryptography libraries are also still in C. Bindings to C code in Swift and Go tend to require dynamic linking. Small libraries like BearSSL were meant to be embedded. (seeing the pattern, yet?) Nim wins this point with no contest.
3. Portability: Nim wins hands down again for being the most portable of the statically typed languages with high level syntax. Go and cgo will port if it's absolutely required, it's C interop is mediocre, and it gets hung up on large file sizes for mobile targets. Swift support on Android is a gray area, and Crystal really only works on Linux, and is single threaded (not on Linux) and without mature network IO support when targeted to Windows. Without large corporate sponsors and bigger uptake, there's only so much that Crystal team has been able to achieve. Like Nim, they've done a lot with a little. When it comes to portability and platform support, Team Nim has done a lot more.
4. Single developer / small team unfair advantage: this requires a high level language. Nim looks like a scripting language but has an advanced memory management system to draw from today (ORC) while being able to prototype with a choice of GC's. Crystal's Ruby like syntax is equally seductive, but its ecosystem is a shadow compared to the work that's gone into Nim. Having said that, Nim does produce concise, highly readable code. Other developers won't have too much of an issue deciphering Nim's code, unlike a lot of Rust I've seen in the blockchain space that has the language's kitchen sink thrown at every file and feature. Go is a close runner-up, but isn't a tie due to it's second-rate C interop. If you don't have to deviate from the stdlib, it's great. But that's not my reality. I also don't have a big need for CSP, and easy fast concurrency for HTTP and gRPC based middlewares is a major reason people reach for Go.
5. Typing: this is more to address criticism I've seen leveled at Nim's type system for being fragile for complex use-cases, but I think this is way overblown. I find Nim's typing to be flexible, intuitive and easy to reason about. Typing is like habanero sauce - a little bit goes a long way.
6. Speed: Nim spits out C. Sure, it can be 2x "slower" than C if your code is naively written. This is just fine, and naive code is likely still faster to execute than Go. It's a non-issue, and making up for any tiny advantage another language might pull over Nim in speed will quickly be crushed by lower memory and file size footprint from Nim. I think in time with v2 we'll see great things from ORC.

Runners up: Dart, Rust, & Zig.

• As an AOT compiled language with a mature set of integrated tooling, Dart has a lot of things going for it, including a decent UX framework in Flutter. It is seamlessly portable across desktop and mobile apps, and just might be the easiest and most shockingly simple for cross-platform development. It still has a runtime penalty, is still niche, and speedwise falls somewhere between Golang and V8. Dart is a very close runner up for having that fast glue-ware language with a very reasonable looking FFI for consuming libraries. Unfortunately, it doesn't extern functions like Rust or Go, isn't C like Nim, and would necessitate a higher level API as an embedded lib, which is a deal breaker. Like Java and Javascript, brackets heavy syntax can be a bit annoying, nobody is using it for embedded or back-end code, and security vulnerabilities might be lurking around that aren't exposed with typical fuzzers or analytical tools. At the end of the day, this statically compiled language still has most of the drawbacks of Typescript or C#.
• Rust: I'm not afraid to admit it - Rust is just too complicated for use as an aging and cranky solo developer. Aside from Dalek and some of the more bleeding edge cryptography for Zero Knowledge Proofs, there isn't a lot of reason to defer to Rust as an obvious choice outside of the usual focus on memory safety that dominates public opinion. As someone who actually hates to code for coding's sake and wants to Get Sh!t Done, Rust is just antithetical to my relationship with computers. I don't want to have to think about the language, and that's all Rust wants you to think about. I see a lot of people attracted to Rust where there is no shortage of funding and developer focus tends to be competitive, often leaning towards academic purity over solving pressing customer facing problems. These aren't luxuries I have and, to be frank, is not the profile of developer that one should hire for in an early stage startup. Nim gets out of the way, and stays out of the way.
• Zig: instant love affair. This is the better C we've been waiting for. D tried, got really close, and failed to kill C. Like D and Nim, Zig is truly compatible with C and will compile C code directly, since its compiler is actually written in C++. Due to the breadth of software that's still written and lightly maintained in C, I think many C devs and new devs faced with legacy C will prefer to port their software to Zig over porting it to Rust, which is existentially different in both approach and complexity. Zig's big hat trick is that it will permit maintainers to port their applications over one file at a time and compile new and legacy code at the same time using the same tool chain. Despite this, Zig is a young language without much documentation for newbies, and essentially markets itself to C programmers. It has yet to reach a v1.0 and breaking changes are still in the cards as the Zig team knock down important milestones, the most recent being a self-hosted compiler.

Brief note on WASM compatibility.

I'm not keen on WASM, because I'm not that keen on web browsers / webview remaining the dominant UX for another 20 years. All of these languages have fine compatibility with WASM, but despite my wishes, Nim goes a step further and compiles directly to Javascript. This means that performant C intermediate code could be targeted to WASM, and shim code can be exported to Javascript - all without leaving the ergonomics of Nim. Potentially that means no RPC middleware required in web uses for me, which is very, very attractive.

Considered, didn't make the cut:

• JVM is dead to me. I was a beta user for Java 1.0 and spent 10 years working exclusively with Java and CORBA middlewares. Kotlin Native just isn't working well out of the box, and is a long way off of being supported well on a wide range of targets. I like Kotlin's syntax, but the weight of the JVM's bags are just too heavy. I also despise being reliant on an IDE to be reasonably productive, tending to prefer vim for editing. JNI and the JVM's largesse isn't aging particularly well. It will die with my generation's retirement.
• Pony. Oh, for years I have wanted to mix it up on the range with Pony and its integrated support for Actor based programming, no-compromise correctness, and focus on low latency performance. Pony's designer, Sylvan Clebsch, is from Wall Street and worked in high performance financial systems. He also happens to be an insanely gifted computer scientist who now works at Microsoft Research. Pony has probably the best type system in any language existing today, giving even Haskell a run for its money. It's built on top of LLVM and has a lot of potential, but not many takers. The community's head maintainer does a herculean job of keeping this project afloat, but it's not been well received, with Rust gaining a lot of Pony's early mindshare in these recent years.
• D-lang. Ooph. I think the D folks fell into the same trap as Rust in trying to support every esoteric programming fetish stemming from toy-project and side-hustle demands among People Who Code. Instead of dominating the market, they were painted as a bunch of weirdos and kicked to the curb of obscurity. There are two competing compilers, competing coding paradigms, and a boat load of great documentation that will yield a high quality and fast-as-C program. In my not-so-humble opinion, D developers are some of the best programmers alive. It actually has strong support on the Bitcoin side, since one of the node implementations is written in D, and that lead developer happens also to be in my neck of the woods. C interop is as good as C. Here's the brutal truth: I can't find anyone under the age of 41 in my field to say a single positive thing about D, and I've been haunting the forum there since .. well, since a very long time. Nim aside, I think Camp D should've won the language wars quite a long time ago, and Rust should've died as Graydon's fever dream before accepting that cushy job at Apple to work on Swift. What can ya do?
• OCaml, Haskell, F#, and other weirdos: I personally can't do it. I tend to not like functional programming languages. I want to, but my brain lacks the plasticity to relearn everything. Younger developers feel inferior for not being fluent in Rust. I feel inferior for not being fluent in OCaml. I am a skill-less and soul-less hack. A peasant. I weep

Final thoughts.

Coming back to Nim makes me kick myself a bit now in hindsight for being too timid to adopt it earlier. Sticking with Nim long term has no visible downsides. Two bonus points: 2) As of today, Status is one its primary corporate backers, having ported an Ethereum node fully to Nim, and open-sourcing the whole thing with good utilities also for EVM based tools. Those are solid wins for my use-cases. 2) Nim's v2 roadmap appears to be focused on embedded applications. For a tiny company that's working with cryptography and wants to safely target as many devices and environments as possible, this is a great thing. Mature tooling like valgrind and an array of fuzzers and other security based utilities work well on Nim binaries.

Hopefully this is useful to someone. Look forward to posting a bit more on the forum here in the months to come. cheers