Introducing bchd - bitcoin cash in Go

https://github.com/gcash

I've been wanting to see someone port the btcd full node implementation over to Bitcoin Cash for over a year now since it's a really well written implementation and Go is my favorite programming language. Doing so, however, is a pretty daunting task as you not only need to implement four different hardforks which change the consensus rules, but (ideally) you'd need to completely rip segwit out of the codebase and segwit affects around 5,000 lines of consensus code across more than 70 files. So I didn't really want to be the one to do it. But since it's been over a year and nobody else has done so I figured I'm probably familiar enough with both the codebase and Bitcoin Cash consensus rules that I could do it in a reasonable amount of time. It took about two weekends of working around the clock to make it happen. I wasn't particularly planning on announcing it any time soon as I'd like to have at least have beta software people can use first, but I blabbed about it on Reddit and so now it probably makes sense to write a blog post about it. So here's a brief introduction to btcd and the items that I'd like to see on it's roadmap.

What is btcd?

Btcd is a Bitcoin full node implementation written in Go (golang) that has been around since 2013. It was mostly written by Dave Collins working for a company name Conformal. Since 2013 it has seen open source contributions from 83 developers and is currently maintained by Roasbeef as part of the broader lightning network project.

The codebase is one of the best designed and well written Bitcoin codebases. The core of the software suite is a blockchain daemon (btcd) which is completely segregated from the wallet (btcwallet) code providing a clear separation of concerns and making it easier to maintain. Further each package inside btcd is very well organized to the point where it also can be used a library for other Bitcoin apps. For example, in OpenBazaar we use btcd as a library and make extensive use of several of its packages. Our spvwallet code also uses large parts of btcd.

Why does Bitcoin Cash need another full node?

This is a very good question! The first part of the answer has to do with the language it’s written in. While Go might not be quite as fast as C++, it still is a pretty fast language and it’s much easier to read, write, and compile than C++. This is critically important as we want to attract as many developers into the Bitcoin Cash space as possible and the existing C++ implementations can be extremely intimidating to newcomers.

By contrast Go and the bchd codebase are very accessible and friendly to new developers. It offers great concurrency primitives and is really ideal for these types of highly concurrent networking applications. As such the number of developers capable of contributing to bchd is likely to be greater than the number of developers capable of contributing to the C++ implementations. Further, the bchd codebase is far more hackable than the C++ implementations which makes it a great choice for apps which build on top of Bitcoin Cash. As mentioned above, OpenBazaar is one of these apps that used parts of btcd within its own application.

Finally, as the recent inflation bug in Core has demonstrated, there is a critical need for implementation diversity. When everyone either uses the same codebase or forks derived from that codebase, it introduces a systemic risk into system. A completely different codebase, in a different language, written from the ground up can help mitigate those risks.

A high quality blockchain server

Presently Bitcoin Cash (as well as Bitcoin) is severely lacking a high quality, reliable indexing blockchain server. The current offerings are not very well done and leave a lot to be desired.

Insight Bitpay’s insight server is probably the most popular blockchain server, but it’s far from great (we're currently building on it in OpenBazaar). For the most part it isn’t very reliable. It will crash from time to time and there are reports of scalability issues. Further the API isn’t that great and regularly fails to push data over websockets. Finally, it does not serve any SPV proof which means if you want to build a wallet on top of the API it must be 100% server trusting.

Electrum The server is written in Python which is a fairly poor choice language for a server that needs maximum performance. Undoubtedly the server will have scalability issues as the network grows. Further while it does offer SPV functionality it has a poorly designed API. The fact that the only application using the Electrum server is the Electrum wallet is likely a testament to how poor the API is. Developers need something better.

Libbitcoin At one point people had high hopes that Libbitcoin would be that high powered blockchain server that was desperately needed, but it never really panned out. We used Libbitcoin in OpenBazaar a few years ago and at least at the time it crashed multiple times a day and frequently needed to be completely resynced. It got to the point where everyone else running libbitcoin servers just shut them down and used ours as we were the only ones with the patience to try to keep them running. And finally the choice of ZMQ for the API is just awful and no developer should be forced to add a massive C dependency to their application in order to use it.

Bchd So one of my primary goals with the bchd project will be to provide a high powered indexing blockchain server that will act as a fast and reliable backend for Bitcoin Cash applications with a great API. The server already supports an optional --txindex and --addrindex which should be enough for most application needs. It also has a really nice Indexer interface and IndexManager than can be used to implement any other indexes that are needed.

Additionally, I'd like to add a more modern gRPC API which uses a binary format over HTTP/2 and has streaming capabilities in addition to (or as a replacement?) for the current JSON-RPC API. gRPC has a plugin option to spawn a REST reverse proxy for applications which need a JSON API.

Finally, we will implement all the API calls necessary to serve fully functional SPV wallets.

New features

The design of the codebase coupled with the ease of use of Go makes it relatively easy for us to experiment with new features that would take a long time to get into the C++ implementations. A sampling of those I’d like to work on include:

Client Side Block Filtering This is a feature that we get for free out of the box as the work has already been done by Roasbeef and other btcd developers. Client side filtering (bips 157 and 158) is a technique whereby one can build a p2p SPV client with better privacy than current bloom filtering SPV wallets. To support these wallets full nodes must maintain a filter index to serve to lite clients. Bchd maintains this filter index by default meaning that this SPV functionality is already available on the Bitcoin Cash network to any developers who want to build a wallet which makes use of it. (This is running on service bit seven).

Further, the bchwallet has code to use the neutrino backend which implements the client side filtering SPV functionality. We just need to port it over.

QUIC QUIC is a new transport which was developed by Google to replace TCP. It sees sizable performance improvements on the worst type of connections ― those with high packet loss and long round trip time. Precisely the type of connections we have in the Bitcoin Cash network. Not only does QUIC have the potential to speed up the network, but it would give us encrypted and authenticated connections (using TLS 1.3) by default. This not only improves the security of the network but finally makes it possible to connect your mobile SPV wallet to your home node using an encrypted and authenticated connection.

I'd like to experiment with QUIC in bchd by enabling it as a transport alongside TCP. When connecting to other bchd nodes the software will prioritize using QUIC.

Fast Sync There are several ways fast sync can be done but the easiest way is to just checkpoint the UTXO set and put the UTXO set up on AWS (or better yet, IPFS). We can then use something like ECMH to verify the hash of the UTXO against our checkpoint and get started right away. This is relatively low difficulty and we can get it in fairly quickly and iterate from there.

New Mining RPC Getblocktemplate has worn out its welcome. I’ve personally done a little work on a new mining RPC but so have Matt Corallo and the Core developers. Most likely I’d implement Corallo's betterhash RPCs (or possibly some gRPC variant) into bchd so we can experiment with it and other pool designs.

State of the codebase

The following has already been done.

✔ Rebranding: All references to BTC and Bitcoin have been changed to BCH and Bitcoin Cash.

✔ Remove Segwit: Every last line of related to segwit has been completely stripped from the codebase.

✔ Implement August 1st, 2017 hardfork

✔ Implement November 13th, 2017 hardfork

✔ Implement May 15th, 2018 hardfork

✔ Implement November 15th, 2018 hardfork (currently open in a pull request)

The following still needs to be done before a beta version can be released:

⧠ Create UTXO cache: A major issue with the codebase is that it lacks a UTXO cache making the initial chain sync dreadfully slow. There is an open PR in btcd but it has bugs in it and isn’t ready to merge.

⧠ Implement pruned mode: Another major feature that is missing is pruned mode which will need to be implemented to provide feature parity with other implementations.

⧠ Port btcwallet/neutrino: The btcwallet codebase still needs to be ported over and made to use the bchd backend. Likewise with the neutrino backend.

⧠ Switch to libsecp256k1: Most likely we’ll have to use the C bindings for libsecp256k1 to get optimal signature validation performance. We can probably get ECMH this way as well. If you're interested in working on it drop by the Github repo. The more the better!