Engineering Update: BuildKit 0.9 and Docker Buildx 0.6 Releases

On July 16th we released BuildKit 0.9.0, Docker Buildx 0.6.0, Dockerfile 1.3.0 and Dockerfile 1.3.0-labs. These releases come with bug fixes, feature-parity improvements, refactoring and also new features.

Dockerfile new features

Installation

There is no installation needed: BuildKit supports loading frontends dynamically from container images. Images for Dockerfile frontends are available at docker/dockerfile repository.

To use the external frontend, the first line of your Dockerfile needs to be pointing to the specific image you want to use:

# syntax=docker/dockerfile:1.3

More info: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md

RUN command now supports --network flag

RUN command supports --network flag for requesting a specific type of network conditions (host, none, default). This allows the user to disable all networking during a docker build to ensure no network is used, or opt-in to explicitly using the hosts network, which requires a specific flag to be set before this works.

--network=host requires allowing network.host entitlement. This feature was previously only available on labs channel:

# syntax=docker/dockerfile:1.3
FROM python:3.6
ADD mypackage.tgz wheelz/
RUN --network=none pipe install --find-links wheels mypackage

RUN --mount flag variable expansion

Values for RUN --mount flag now support variable expansion, except for the from field:

# syntax=docker/dockerfile:1.3
FROM golang
...
ARG GO_CACHE_DIR=/root/.cache/go-build
RUN --mount=type=cache,target=$GO_CACHE_DIR go build ...

Here-document syntax

RUN and COPY commands now support Here-document syntax allowing writing multiline inline scripts and files (labs channel) without lots of && and \ characters:

Before:

# syntax=docker/dockerfile:1.3
FROM debian
RUN apt-get update \
  && apt-get install -y vim

With new Here-document syntax:

# syntax=docker/dockerfile:1.3-labs
FROM debian
RUN <<eot bash
  apt-get update
  apt-get install -y vim
eot

In COPY commands source parameters can be replaced with here-doc indicators. Regular here-doc variable expansion and tab stripping rules apply:

# syntax=docker/dockerfile:1.3-labs
FROM alpine
ARG FOO=bar
COPY <<-eot /app/foo
  hello ${FOO}
eot

More info on BuildKit repository.

OpenTracing providers replaced with support for OpenTelemetry

OpenTelemetry has superseded OpenTracing. The API is quite similar but additional collector endpoints should allow forwarding traces from client or container in the future.

JAEGER_TRACE env is supported like before. OTEL collector is also supported both via grpc and HTTP protos.

This is also the first time cross-process tracing from Buildx is available:

# create jaeger container
$ docker run -d --name jaeger \
  -p 6831:6831/udp -p 16686:16686 \
  jaegertracing/all-in-one

# create builder with JAEGER_TRACE env
$ docker buildx create \
  --name builder \
  --driver docker-container \
  --driver-opt network=host \
  --driver-opt env.JAEGER_TRACE=localhost:6831 \
  --use

# open Jaeger UI at http://localhost:16686/ and see results as shown below

Resource limiting

Users can now limit the parallelism of the BuildKit solver, Buildkitd config now allows max-parallelism for limiting the parallelism of the BuildKit solver, which is particularly useful for low-powered machines.

Here is an example if you want to do this with GitHub Actions:

We are also now limiting TCP connections to 4 per registry with an additional connection not used for layer pulls and pushes. This limitation will be able to manage TCP connection per host to avoid your build being stuck while pulling images. The additional connection is used for metadata requests (image config retrieval) to enhance the overall build time.

GitHub Actions cache backend (experimental)

We have released a new experimental GitHub cache backend to speed up Docker builds in GitHub actions.

This solves a previously inefficient method which reuses the build cache by using official actions/cache action together with local cache exporter/importer, which is inefficient as the cache needs to be saved/loaded every time and tracking does not happen per blob.

To start using this experimental cache, use our example on Build Push Action repository.

Git improvements

Default branch name is now detected correctly from remote when using the Git context.

Support for subdirectories when building from Git source is now released:

$ docker buildx build git://github.com/repo:path/to/subapp

SSH agent is automatically forwarded when building from SSH Git URL:

$ docker buildx build [email protected]:myrepo.git

New platforms and QEMU update

• Risc-V (buildkitd, buildctl, buildx, frontends)
• Windows ARM64 (buildctl, buildx)
• MacOS ARM64 (buildctl)

Also embedded QEMU emulators have been updated to v6.0.0 and provide emulation for additional platforms.

--metadata-file for buildctl and buildx

New --metadata-file flag has been added to build and bake command for buildx that allows saving build result metadata in JSON format:

$ docker buildx build --output type=docker --metadata-file ./metadata.json .

{
  "containerimage.config.digest": "sha256:d8b8b4f781520aeafedb5a88ff50fbb625cfebad87e392794f1e26a724a2f22a",
  "containerimage.digest": "sha256:868f04872380274dcf8528222e84dc66702394de80889e51c87a14126ea9ff6a"
}

Buildctl also allows --metadata-file flag to output build metadata.

Docker buildx image

Buildx binaries can now be accessed through buildx-bin Docker image to build able to use buildx inside a Docker image in your CI for example. Here is how to use buildx inside a Dockerfile:

FROM docker
COPY --from=docker/buildx-bin:latest /buildx /usr/libexec/docker/cli-plugins/docker-buildx
RUN docker buildx version

Other changes

For the complete list of changes, see the official release notes: