The Problem: cgo, cross-compilation, and Bazel

The Photos system stores, processes, and serves customer-supplied images, as described in The Making of Pages, part 2. It is implemented in Go and performs its image processing tasks using the high-performing and feature-rich ImageMagick, via the gographics/imagick package.

This worked, but it had a few issues:

• ImageMagick of a specific version must be installed on developer workstations and our CI VMs.

• Cross-compilation does not work, because no one knows how to set up a cross-compilation toolchain for C++.

Then we migrated to Bazel so that we could build container images, automate our code generation steps & dependency management, and speed up builds with a team-wide cache and safe incremental builds. gographics/imagick does not build with Bazel out of the box.

Here’s how we added Bazel support and solved the two issues mentioned above.

The Solution: Vagrant & Static linking

Taking a page out of Go’s build philosophy, we can build statically-linked libraries for ImageMagick for MacOS/Linux and check them in. To easily build those libraries for multiple platforms and test that the results work, we use Vagrant and provision the local VMs with Ansible. This assumes a MacOS host.

The entire change can be seen here: yext/imagick 7827fa8f

Working through it:

1. Vagrantfile configures three different Linux VMs, used for building the ImageMagick library and testing the results in two different environments: one mimicking a developer workstation and another mimicking production. Vagrant provisions the “developer workstation” using vagrant-dev-playbook.yml, an Ansible playbook.

2. download-build-imagemagick.sh is run from both the MacOS host and the Linux builder VM to download and build static ImageMagick libraries.

3. imagick/magick_wand_test.go includes a unit test that validates the set of delegates (image formats) supported by the library, providing confidence that it was built properly.

4. imagick/magick_wand.go has an update to the #cgo directives, which pulls in the prebuilt libraries instead of using pkg-config.

5. libs/BUILD.bazel contains the cc_library Bazel target, referenced by the go_library in imagick/BUILD.bazel.

In the end, I was able to build on MacOS and Linux, run the tests, and cross-compile. Nirvana achieved.