My Go Executable Files Are Still Getting Larger (What's New in 2021 and Go 1.16)

Introduction

Two years ago, my article"Why are my Go executable files so large?" showed how to utilize D3 and a tree map visualization to explore the size of executable files produced by the Go compiler.

A few things have changed since, and so an update is in order.

Tooling updates

As presented the first time around, we are using a data pipeline that looks as follows:

1. build a Go executable.
2. use go tool nm -size and apply c++filt on the output.
3. transform the output into a tree using a custom-designed Python script tab2dict.py.
4. transform the tree into a valid input for the D3 tree map visualization using a custom-designed Python script simplify.py.

The source code for the Python scripts is public on GitHub: https://github.com/knz/go-binsize-viz

Since 2019, the Go and C++ compilers produce a larger diversity of symbols; the regular expressions used in tab2dict.py have been adjusted accordingly.

Separately, there was a usability shortcoming in the original implementation: if a Go package contained both some source files (e.g. sql/create.go) and sub-packages (e.g. sql/sem/tree/eval.go), the “own” size of the package and that of its sub-packages were appearing side-by-side in the visualization, instead of “inside” each other. This was confusing because the (human) explorer naturally expects a hierarchical view between these two values.

This shortcoming has also been corrected.

Surprising finding: “dark” file usage

The sum of the sizes reported by go tool nm does not add up to the final size of the Go executable.

For example, in the CockroachDB 20.2.7 binary:

• the file occuppies 211694984 bytes (202MiB) on disk;
• however, the sum of symbol sizes adds up to 118928245 bytes (113MB).
• there is a gap of 92766739 bytes (88MiB) missing, or ~44% unaccounted for.

At first I suspected that this size was occupied by the symbol table itself, or the debugging information. To check this, we can use strip to remove the symbol table and observe the difference. Alas:

• the stripped executable size is 190680384 bytes (182MiB) on disk;
• so there is still a gap of ~68MiB, or ~34% non-symtable data that is unaccounted for.

At this time, I do not have a satisfying explanation for this “dark” file usage.

We can see how this dark file usage has evolved throughout the growth of CockroachDB:

In this table:

• “Exec. size” is the raw size of the executable file, in bytes.
• “Stripped” is the size of the executable after the strip command was applied; i.e. after the symbol table and debugging information was removed.
• “Sum nm -size” is the sum of the advertised sizes of the entries in the symbol table.
• “Symtable sz.” is the estimated size of the symbol table itself, as deducted by taking the difference between the first two sizes. We can see that the v1.0.7 to v2.0.7 executables, as well as v20.1.0, were released pre-stripped.
• “Dark bytes” is the gap between the raw file size and the combined sum of the advertised symbol sizes and the symbol table’s size, in bytes.
• “% dark bytes” is the percentage of the dark bytes relative to the raw file size.

We can see that the dark size percentage was lower than 20% prior to CockroachDB v19.1, and has then been oscillating around 33% of the file size until v21.1. With the upcoming v21.1 release, using Go 1.15, the dark size is reduced to 15% again.

The evolution of pclntab

Situation with Go 1.15 and 1.16: “Oops?”

Using the source code for CockroachDB v21.1-alpha-geb1aa69bc4, we can produce custom builds across Linux and FreeBSD, with both the 1.15 and 1.16 compilers.

What do we see here?

The bytes previously occupied by pclntab are now part of the dark bytes, which are not in the symbol table.

Sure, the Go team can be proud that “pclntab has been reduced to zero”, but the net effect on the excutable size is not so clearly visible!

So much data! For what exactly?

An interesting way to think about the results above is that we now have three parts of a Go executable file that do not really contribute to making a program “work”:

• The symbol table itself (which can be stripped via strip, but is not stripped by default).
• The pclntab, when generated.
• The “dark bytes”, which is byte usage in the raw executable file not accounted for in the symbol table.

How many bytes remain that are “useful”? We can compute this as follows:

• take the stripped executable, to exclude the symbol table and debugging information.
• remove the dark size (stripped exec size, minus size of all the symbols accounted for).
• remove the size of pclntab.
• see how many bytes remain.

This gives us:

To summarize, early on (in Go 1.8 and before) the non-code, non-data part of an executable was less than 40% of the total executable size.

Over time, it has grown to more than 70% of the total executable size.

Even with the new pclntab replacement in Go 1.16, where pclntab is computed at run-time, there is no gain: the data used as input for the computation is stored somewhere in the executable and its total size is larger than the original pclntab even was.

These Go executable files are rather… bloated.

Summary and conclusions

In our original analysis in 2019, we looked at the output of go tool nm -size and drew a tree map representation for it. This helped us detect an anomaly, in the size of a special data structure called runtime.pclntab, which was growing excessively large for no good reason.

After that article was published, the Go team decided to change course and reduce the size of pclntab (by computing it at run-time); so that it is finally absent from binaries produced by Go 1.16.

Alas!

This year, we revisited the analysis and discovered that the symbol table is not complete. There are many bytes in the binary executable that are not accounted for, neither by the announced size of objects in the symbol table, nor by the size of the symbol table itself.

We can call this the “dark file usage” of Go binaries, and it occupies between 15% and 33% of the total file size inside CockroachDB.

Sadly, the removal of pclntab in Go 1.16 actually transferred the payload to the “dark” bytes.

Moreover, if we take a step back, we realize that neither the symbol table, nor pclntab, nor the dark file usage really contribute to the functionality of a program: really, the functionality of a program comes from code+data objects, which are properly listed in the symbol table. We can call these the “non-useful bits” of the binary file.

Even more sadly, the non-useful bits have grown over the course of Go versions. In CockroachDB, they were less than 40% of a raw executable back in v1.0, compiled with Go 1.8, and have grown beyond 70% in v21.1, compiled with Go 1.16.

That’s right! More than two thirds of the file on disk is taken by… bits of dubious value to the software product.

Moreover, consider that these executable files fly around as container images, and/or are copied between VMs in the cloud, thousands of times per day! Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason!