Introduction

Have you ever wanted to have a Go program that ran on multiple platforms from a single binary? I’d always kinda hoped that Go would have “fat” binaries like the old Mac’s used to have when we converted from PowerPC to Intel that let you run on either platform.

So what can we do today if we want to distribute our Go program so that it supports a multitude of Platform/OS combinations from a single executable?

Trusty old shell scripts.

Advertisement

If you are interested in using Go for DevOps work, checkout my book on Amazon: Go For DevOps .

The proceeds for the authors (not the publisher) is going to Doctors Without Borders, a great organization.

Let’s get this out of the way

I actually detest shell scripts for most use cases. They are error prone and testing is a pain. My main use case for shell scripts is running a for loop around a flaky tests to try and get it to trip again.

But like anything, there are exceptions. sh is on every Unix platform and therefore its universal nature makes it perfect for this use.

What caused me to write this?

I recently ran into someone trying to use shell to push data to a service that had SDKs in multiple languages. But that person didn’t want to have different binaries for all the systems they were going to support and didn’t want to have to install other runtimes (Python, .Net, …). This meant doing some reverse engineering on that platform’s REST streams and doing some complicated gymnastics in shell while invoking cURL.

I thought that there must be a better way and cooked this up as a demo. The idea comes from this methodology Google and Facebook use to package up Python programs for distribution.

The Idea

We are going to wrap multiple Go binaries in a shell script and the shell script will detect the OS/Platform and decode the binary we want to the correct location and invoke it. We can pass the arguments to the shell script directly to the binary.

Now this will make the shell script large if you have enough binaries or large programs. An alternate version could have the script download the correct version from GitHub or another location to save space. But for this demo we will just bake the binaries in.

All the code for this demo can be found here: https://github.com/johnsiilver/shellembed

The Wrapping Script

To make this all work, we need a script that can wrap our Go binaries and then unwrap them when invoked.

#!/bin/bash 
# define the tasks that need to be done with the extracted content
process_tar() {
    cd $WORK_DIR"/bin"
    # do something with the extracted content
    eval "./hello_${os}_${arc} $@"
}

This starts by defining a function we are going to call later called process_tar(). That function is going to enter a work directory we will create later and run the eval command on a binary that will be in the work directory, passing it the arguments the script was passed (the $@).

# line number where payload starts
PAYLOAD_LINE=$(awk '/^__PAYLOAD_BEGINS__/ { print NR + 1; exit 0; }' $0)

Next we use awk to scan the script file we are in and record the line number that has __PAYLOAD_BEGINS__. Everything after this will be our Go binaries that have put in a compressed tar format at the end of the shell script.

# Determine the os and arch.
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
  os="linux"
elif [[ "$OSTYPE" == "darwin"* ]]; then
  os="darwin"
else 
  echo "unsupported OS: " + `uname -a` 
  exit 1
fiif [[ `uname -m` == "arm64" ]]; then  
  arc="arm64"
elif [[ `uname -m` == "x86_64" ]]; then 
  arc="amd64"
else 
  echo "unsupported arch: " + `uname -m` 
  exit 1
fi

We want to support Linux and Darwin both running on x86_64 and arm64 processors. We use $OSTYPE and uname to determine our OS and platform. If any other combination is present, we give an error.

# make our directory if it doesn't exist.
mkdir -p /tmp/helloshell
WORK_DIR=/tmp/helloshell

We then make some work directories where we will dump our extracted tar file to get at our Go executables.

# extract the embedded tar file
tail -n +${PAYLOAD_LINE} $0 | tar -zpx -C $WORK_DIR >/dev/null 2>&1# perform actions with the extracted content
process_tar $@exit 0
__PAYLOAD_BEGINS__

We then make some work directories where we will dump our extracted tar file to get at our Go executables.

We then untar our Go executable and execute our process_tar() function from above.

Now we have a script that can untar some Go binaries and run them if we support the platform. But how do we get the Go binaries into the script?

Build For Platform Script

Here is a small script that builds our binaries for each OS/Platform combination we wanted and then embeds them in a file called run.sh. This file is a copy of hello.sh with the Go binaries embedded as compressed tar files.

This builder script will be called build_for_platform.sh

#!/bin/bashexport GOOS=linux 
export GOARCH=amd64 
go build -o bin/hello_linux_amd64export GOOS=linux 
export GOARCH=arm64 
go build -o bin/hello_linux_arm64export GOOS=darwin
export GOARCH=amd64 
go build -o bin/hello_darwin_amd64export GOOS=darwin
export GOARCH=arm64 
go build -o bin/hello_darwin_arm64

This first section simply builds our Go binaries in a directory called bin/ indexing them by OS and platform.

tar -czvf hello.tar.gz bin/

This next section creates our tarball of bin/

cp hello.sh run.sh
cat hello.tar.gz >> run.sh

Now we create our copy of hello.sh as run.sh and append our tarball to the end of run.sh .

rm hello.tar.gz
rm -rf bin/

Finally, we do some cleanup.

Drumroll, if you will

You can now simply run:

./build_for_platform.sh

Which will create your run.sh file.

Now let’s execute our run.sh and pass it an argument (which is a name it will print out):

./run.sh John

This will output something like this (output depends on the platform):

hello John from: darwin/arm64

Conclusion

As you can see from above, it is possible to use shell scripts to have a universal binary for Go across all supported platforms. If you wish to keep the shell script small, you could have it simply download the binary you need and call with the same arguments.

But this is a good way when you don’t want to depend on the internet.

If you’d like to see more of my writing, you should check out my book on Amazon: Go For DevOps

You can also find more writings on various topics here on medium or my blog at golangsre.com. And if you want to learn Go from some interactive video lessons, see my free site golangbasics.com.