A tale of two toolchains and glibc

A tale of two toolchains and glibc

Over the past few years, the LLVM toolchain has seen increasing development and adoption alongside the older, more established GNU toolchain. The emergence of this new two major toolchain world is bringing challenges and questions for projects that need to support both, in particular the GNU C library (glibc), which only supports GCC.

Is it worth it to fix glibc (and other projects which support only GCC) to build with LLVM? Is it better to just replace them with alternatives already supporting LLVM? Is it best to use both GCC and LLVM, each for their respective supported projects?

This post is an exploration starting from these questions but does not attempt to give any definite answers. The intent here is to not be divisive and controversial, but to raise awareness by describing parts of the current status-quo and to encourage collaboration. The obvious elephant in the room, licensing, is left out despite being a very important topic.

The toolchain support burden

Distributions which use or allow LLVM/Clang to be used as main compiler (Yocto/OE, Gentoo/ChromeOS, OpenMandriva, Alpine and others, maybe even Android can be counted here) are forced to use different libcs or maintain a GCC toolchain just for building a small subset of packages at the center of which is typically glibc. This is not ideal considering the need for bug fixes and new HW feature support, both toolchains requiring active maintenance.

Dropping GCC and downstream-patching individual projects to support Clang is hard because patches break on upgrades (unless you fork and never merge back) and can introduce bugs or regressions, especially in the case of glibc where the diff is big and complex. Small patches like configuration, cross-compilaton fixes or bugfix backports might be acceptable downstream, but keeping something like LLVM support downstream is unfeasible, so first-class support and careful testing to avoid regressions should ideally happen in the upstream project.

The glibc problem

When building a system with LLVM/Clang, the most common problem is the GNU C library which is widely used and depended upon. The C library in general is part of the base OS definition and some other important projects like systemd can only be built specifically with the GNU implementation of the standard lib. Glibc is probably the most problematic to build with LLVM/Clang (at least of the GNU projects) due to its reliance on GNU toolchain extensions or specific non-standard GCC behaviours or bugs. This is understandable given the age and purpose of the project.

Glibc built by LLVM should be achievable if we consider the successful effort to add first-class Clang support to the Linux kernel which is also a complex and historically tied-to GCC project. The problem is: at what cost and when does the glibc + LLVM cost exceed the time & effort required to use a different libc or maintain a GCC toolchain only for glibc? At least in theory swapping C libraries should be much easier than swapping Linux for another kernel, so we can assume that cost is lower. Same with keeping GCC around.

There are some "clangify" branches in upstream glibc (e.g. shebs/clangify), but they are based on 2.26-2.29 versions and development seems to have stalled in the last few years.

Divergent behaviours of toolchains

Sometimes Clang and GCC can function as drop-in replacements to one another, but this is not always the case. Obvious examples are the different build flags or extensions that require special Clang enable flags. A more subtle example is that Clang "claims" to support -std=gnu99 while it does not support nested functions which is part of gnu99. Things get messy when there is no explicit standard to govern a behavior and projects may depend on whatever happens to work in a toolchain, typically GCC. For eg. the GNU assembler is more permissive than the Clang assembler which, among other things, rejects ambiguous instructions. It might not be entirely clear in which project is the "bug".

Project support code

Due to LLVM vs GCC differences like the above, various projects end up having to #ifdef toolchain specific code which can be a source of bugs in addition to making the code uglier in general. This is very nicely reported in a presentation by OpenMandriva developer Bernhard Rosenkränzer at EuroLLVM 2019. The examples are sad and comic at the same time. As context for the following example, Clang defines __GNUC__ == 4.

#if __GNUC__ >= 5
do_something_sane();
#else
assume_the_compiler_sucks();
#endif

Project codebase complexity

Like many other projects with a long history, the glibc codebase is also complex and this can become an impediment to contributions wanting to fix Clang support. For a specific example, see this patch from 2014 which tried to replace nested function declarations in regcomp.c. That specific source file is shared with gnulib which means it requires two rounds of review from the respective projects and unfortunately the files were out of sync between projects so the patch was NAK'ed until the sources could be put in sync. The patch author decided to work on something else and so this specific nested function declaration remains unfixed until today.

Progress is made

Even without a big concentrated effort like the Linux kernel had, LLVM/Clang support is slowly improving in glibc as a result of a kind of "grass-roots" movement. A recent admirable effort of persistence from Fangrui Song added support for linking Glibc with LLD, the LLVM linker, starting with the soon-to-be-released LLVM 13 and glibc 2.35 which will be released in a few months. Fangrui Song also wrote a blog post describing his experience which is a very good read for anyone interested in the subject.

Is Glibc + LLVM worth it?

Anything is possible in the future. What is certain is that LLVM adoption is growing in both FOSS and the software industry in general and there are alternatives to projects which are too tied up with GCC specifics, so it is no surprise choices are made to replace them.

For example elfutils which is an established alternative to GNU binutils will also support building with Clang in its next release (v0.186). It is missing a linker, but it can be provided by the LLVM Core. LLVM Core itself provides alternatives to GNU binutils while at the same time working to improve GNU binutils compatibility and add missing features.

There are quite a few C standard library implementations which are not as dependent on GCC as glibc. There's musl which is (maybe too strictly) standards-compliant, quite a number of smaller embedded-focused libc's like newlib or uClibc, there is also bionic which mostly aims at supporting Android, there is a libc written in Rust and even the LLVM project is creating their own C standard library. Given so many alternatives, it should be no surprise that LLVM/Clang support work is scarce in glibc.

A problem the alternative C libraries have is that feature development primarily happens for glibc with HW vendors understandably targeting the most popular and widely used C library.

Conclusion

We are living in interesting times in this area of the software stack after so many years of GCC toolchain dominance. Having to choose between high-quality fundamental projects is awesome. The newer LLVM/Clang is developing towards becoming a complete alternative more focused on standards-compliance, but obviously GCC will continue to be very popular and will not go away anytime soon. The LLVM C standard library in particular should need quite a lot of time before it becomes production ready.

In a two-toolchain dominated world, projects are adding support for the up and coming LLVM and, at least for now, distribution vendors and (usually embedded) system integrators end up maintaining both toolchains or finding alternatives to toolchain-incompatible projects with different trade-offs.

An argument could be made that perhaps the C language itself is becoming less relevant and alternatives like Rust where the GCC vs LLVM support situation is currently reversed would become what is considered a "standard" toolchain, if for example the Rust std library is made libc independent because currently the libc (musl/glibc) is the OS interface for the Rust std lib. However this is highly speculative and far in the future - it is more likely we will have glibc with LLVM support or LLVM libc by then. :)