Art by Egon Elbre at https://github.com/egonelbre/gophers

I JUMP between languages and when I come back to one I have not used in a while there are certain things I have not forgotten while other things are easier to forget. This is a look at things which you may get confused about in Go because they have changed a lot over the last few years.

Thematically story: Refreshing Your Swift and Cocoa Programming Skills

Here are some of the topics I like to cover:

• GOROOT and GOPATH environment variables. Are they even needed anymore?
• Modules and packages
• Command line tools — Building, fetching dependencies and running tests
• Gotchas when using pointers
• Logging
• Error handling
• Enums
• Embedding data files in Go binaries

Difference Between GOROOT and GOPATH and Do We Need them Anymore?

For some reason, I always confuse Go environment variables with each other, and I keep forgetting whether I require them at all since modules got introduced.

• GOROOT - Location of your Go SDK. Useful if you have multiple versions of Go installed.
• GOPATH - Root of your personal workspace. Defaults to ~/go and Linux and macOS.

You can set these variables yourself. On my Mac GOROOT is not set to anything and Go works fine. You don't need GOPATH to point to the directory where you develop your code. I put my Go code in ~/Dev/go, but you need a place to install third-party go tools and packages. That is where GOPATH comes in. I set my GOPATH to ~/.go in :

# Added to my ~/.config/fish/config.fish
set -x GOPATH $HOME/.go

I put a dot in front to make it invisible (only works on Unix systems). The reason is that you rarely if ever need to go into the GOPATH directory, as you keep your own code separate. You can look contents with the tree command.

❯ tree -L 1 $GOPATH
~/.go
├── bin
├── pkg
└── src

You should add $GOPATH/bin to your PATH to make sure Go binaries can be discovered by your shell and run. Here is how I configure my fish shell to find go binaries:

# To run home-made go commands and installed go programs
set -x PATH $PATH /usr/local/go/bin $GOPATH/bin

Command Line Tools and Modules in Go v1.18

The introduction of modules in Go changed a lot about how we use command line tools and environmental variables in Go. It reduced the importance of environment variables such as GOPATH. Let us look at how I build different parts of my rocket Go project. To create this project, I would do the following:

❯ cd $GOPATH/src
❯ mkdir rocket
❯ cd rocket
❯ go mod init github.com/ordovician/rocket

Once populated with source code files, it will look like the listing below:

❯ tree -L 2 rocket/
rocket/
├── README.md
├── go.mod
├── go.sum
├── cmd
│   ├── launcher
│   └── server
├── engine
│   ├── engine.go
│   ├── engine_example_test.go
│   ├── merlin_engine.go
│   └── rutherford_engine.go
├── math
│   └── math.go
├── physics
│   ├── equations.go
│   ├── motion.go
│   ├── rigidbody.go
│   └── rigidbody_test.go
├── propulsion.go
├── stagedrocket.go
├── stagedrocket_test.go
├── tank
│   ├── flexitank.go
│   ├── tank.go
│   └── tanks_test.go
├── types.go
└── util_test.go

The source code at the top level such as propulsion.go and stagedrocket.go belong to the rocket package. That is reflected in where they are located in the directory hierarchy as well as the source code. The start of my stagedrocket.go file looks like this:

package rocket

import (
	. "github.com/ordovician/rocket/engine"
	. "github.com/ordovician/rocket/physics"
	. "github.com/ordovician/rocket/tank"
)

type MultiStaged struct {
	payload Rocket
	Propulsion
}

While each subdirectory under rocket defines different packages. Here is from the start of the flexitank.go file:

package tank

import (
	. "github.com/ordovician/rocket/physics"
)

// A tank with flexible size
type FlexiTank struct {
	DryMass    Kg
	TotalMass  Kg
	propellant Kg
}

There is an overlap between modules and packages. github.com/ordovician/rocket is a module containing the rocket package. But that is not the only package it contains. It also contains the physics, tank and math packages for instance.

Command Line Tools in cmd

The rocket/cmd directory is interesting because each contain files to create an executable. That requires the main function to be present. However, you cannot have a main function in any package. It needs to be in the main package. That is why each command we are building has a separate directory under rocket/cmd.

Building Packages and Command Line Tools

We can build whole packages, individual files, sub packages or command line tools with the go build command. When creating a module you specify a name for it which should reflect where you will put that module online. For instance, my rocket module is named github.com/ordovician/rocket with the go mod init command because that is the git repository where I will store the package. Because that is the actual name of your module, typically use that when building, running or testing. You would write:

❯ go build github.com/ordovician/rocket

Note that this only works when you are in a directory with a go.mod file which actually defines this module. This build command doesn't build packages which are not rocket such as the command line tools we have in the cmd sub directory. To build a specific command, we have to specify the full package path. The same path you would use when importing a Go package:

❯ go build github.com/ordovician/rocket/cmd/server

Of course, it might be awkward to write the full path like this every time. Fortunately, we can use relative paths. You have to start with ./ because Go will treat cmd/server or /cmd/server as a package name and not a locally defined package.

❯ go build .              # build rocket package
❯ go build ./engine/      # build engine package
❯ go build ./cmd/server   # build server binary

This approach also works for the test and run commands as well. Let us look at running tests defined in _test.go files (shortened output):

❯ go test -v ./tank
=== RUN   TestMakeMediumTank
--- PASS: TestMakeMediumTank (0.00s)
=== RUN   TestMakeLargeTank
--- PASS: TestMakeLargeTank (0.00s)
=== RUN   ExampleMediumTank_Consume
--- PASS: ExampleMediumTank_Consume (0.00s)
PASS
ok  	github.com/ordovician/rocket/tank	(cached)

❯ go test -v .
=== RUN   TestStageSeparation
--- PASS: TestStageSeparation (0.00s)
=== RUN   TestThrustTooLowForGravity
--- PASS: TestThrustTooLowForGravity (0.00s)
=== RUN   TestPlentyPowerful
--- PASS: TestPlentyPowerful (0.00s)
=== RUN   TestCompareWithNewtonMotionEquations
--- PASS: TestCompareWithNewtonMotionEquations (0.00s)
PASS
ok  	github.com/ordovician/rocket	(cached)

To run an individual test, add the -run flag.

❯ go test -v -run TestStageSeparation
=== RUN   TestStageSeparation
--- PASS: TestStageSeparation (0.00s)
PASS
ok  	github.com/ordovician/rocket	0.116s

This also works for sub packages.

❯ go test -v -run TestMakeLargeTank ./tank
=== RUN   TestMakeLargeTank
--- PASS: TestMakeLargeTank (0.00s)
PASS
ok  	github.com/ordovician/rocket/tank	0.164s

We can use the go run command in similar fashion. If you are constantly changing and developing an executable, it is awkward to do a two-step process of first building and then running. Better to combine compiling and running into one step. Here I will show an example from my cryptools module which contains a number of little command for encryption and decryption. In this example, I am generating an encryption and decryption key which is 16 bytes long stored in base64 format.

❯ go run ./cmd/generate -keylen 16 -encoding base64 key.txt

This is equivalent to the following two-step process:

❯ go build ./cmd/generate
❯ ./generate -keylen 16 -encoding base64 key.txt

Installing Go Modules and Tools

In the past, you used go get to get any Go tools you wanted to install. In modern Go we only use go get to install dependencies of a module you develop. So say you are developing a medical application which depends on the GTK GUI toolkit. First, I would create the project:

❯ mkdir medical
❯ cd medical
❯ go mod init github.com/ordovician/medical

Next I add the GTK dependency:

❯ go get github.com/gotk3/gotk3
go: downloading github.com/gotk3/gotk3 v0.6.1
go: added github.com/gotk3/gotk3 v0.6.1

You can see that the go.mod file gets modified to include this dependency:

❯ cat go.mod
module github.com/ordovician/medical

go 1.18

require github.com/gotk3/gotk3 v0.6.1 // indirect

For a lot of older go projects, you will see the README file say that you should write go get to install them, but that no longer applies. Instead, you use go install and you must specify with an @ symbol which version you want. Use @latest to get whatever is the latest. I am old fashion using the TextMate editor, which means I need to install gocode to get command completion in my editor. I do that like this:

❯ go install github.com/stamblerre/gocode@latest

But there are lots of other interesting Go tools you might want to install. Having a debugger is useful. Delve is the more popular one for Go at the moment:

❯ go install github.com/go-delve/delve/cmd/dlv@latest

If you want to install a specific version, you could write:

❯ go install github.com/go-delve/delve/cmd/[email protected]

Removing and Adding Needed Dependencies

You may want to remove or add dependencies depending on what your module actually uses. You can use go mod tidy for that purpose. It removes dependencies not used in any of your source code and actually adds dependencies you need. Find out more with:

❯ go help mod tidy

Gotchas Using Pointers

One of my first mistakes when getting back into Go programming after a break was using pointers wrong, so here is a warning and reminder to help you avoid the same problem. I was specifically building what you call an Arena allocator. It is a way of allocating objects of identical size quickly. It is useful for things like binary trees where the Go garbage collector does not do as well as say Java.

The Arena allocator keeps track of a list of free blocks of memory. Whenever you release a block, it is put back into this list. This code uses Go generics, so it is a useful refresh on Go generics as well:

type Arena[T any] struct {
	blocks Stack[*T]  // NOTE: Stack is a collection I made
}

func (arena *Arena[T]) Alloc() *T {
	if arena.blocks.IsEmpty() {
		var blocks [8]T
		for i, _ := range blocks {
			arena.blocks.Push(&blocks[i])
		}
	}
	b, _ := arena.blocks.Top()
	arena.blocks.Pop()

return b
}

func (arena *Arena[T]) Free(block *T) {
	if block == nil {
		panic("Cannot free nil pointer")
	}

arena.blocks.Push(block)
}

This variant works, but in my initial variant I wrote for-loop wrong:

// DON'T do this, block will be a copy of blocks[i]
var blocks [8]T
for i, block := range blocks {
	arena.blocks.Push(&block)
}

What is wrong with this? Isn’t that just more elegant? The problem here is that Go does not have references. block is not a reference to the element blocks[i] but rather a copy of that element. Hence, &block would be the address of this copy. Because each blocks[i] element got copied into the same memory location upon iteration, I ended up pushing the exact same address onto the arena.blocks stack object. Hence the Alloc method always returned exactly the same object. It was really confusing debugging and finding that whenever I modified one returned object, all the other allocated objects got modified.

Logging

As someone who usually write Julia code, I must say I am not thrilled about logging in Go. Julia comes with built-in logging which has logging levels and which uses LISP like macros which Go doesn’t have to make it very convenient to write logging code. The builtin logging in Go is a bit too bare-bones, while the popular third-party logging tools like Zap isn’t very easy or user-friendly to setup in my humble opinion. The most basic usage of Zap is easy enough:

logger, _ := zap.NewProduction()
defer logger.Sync() // flushes buffer, if any
sugar := logger.Sugar()
sugar.Infow("failed to fetch URL",
  // Structured context as loosely typed key-value pairs.
  "url", url,
  "attempt", 3,
  "backoff", time.Second,
)
sugar.Infof("Failed to fetch URL: %s", url)

But, I find that once I want to do anything more complex I don’t understand much of what the code is doing nor is the documentation explaining well. But for anyone serious about logging building a larger application you should invest in learning Zap as it gives you structured logging. Structured logging is about giving logging data in a structured format, so it can easily be analyzed by tools.

If you would rather not spend a lot of time learning the intricacies of logging technology, then I think the builtin logging framework is better. You just need to learn how to customize it. Here is how I mimic logging levels with the builtin Julia logging system.

var (
	DebugLog *log.Logger
	WarnLog  *log.Logger
	InfoLog  *log.Logger
	ErrorLog *log.Logger
)

// Called first in any package
func init() {
	file, err := os.OpenFile("logs.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0666)
	if err != nil {
		log.Fatal(err)
	}

flags := log.Ldate|log.Ltime|log.Lshortfile

DebugLog = log.New(io.Discard, "DEBUG: ", flags)
	InfoLog  = log.New(io.Discard, "INFO: ", flags)
	WarnLog  = log.New(file, "WARNING: ", flags)
	ErrorLog = log.New(file, "ERROR: ", flags)
}

func main() {
	InfoLog.Println("Starting up")
    WarnLog.Println("Shutting down")
}

With this setup, I am logging to a file called logs.txt. The function for creating a logger looks like this:

func New(out io.Writer, prefix string, flag int) *Logger

The first argument out says where to send the log output. You may notice that for WarnLog and ErrorLog I have set this to file which refers to my logs.txt file, while the others are set to io.Discard. The io.Discard is kind of like dev/null, anything sent there goes away. That is a simple way of disabling logging. So in my example, InfoLog and DebugLog have been disabled.

Error Handling in Go

While handling errors in Go tends to just be about returning an error object, there are a lot of patterns around how to do it which is useful to remember.

In many programming languages such as Java, C# and Python we tend to create entirely new types for each error. Go is all about simple, so different errors in Go tend to be just simple instance values. This is how some common Input/Output errors are defined:

// End of File. When you cannot read more
var EOF = errors.New("EOF")

// ErrUnexpectedEOF means that EOF was encountered in the
// middle of reading a fixed-size block or data structure.
var ErrUnexpectedEOF = errors.New("unexpected EOF")

Keep in mind that errors.New always returns a new object, so You have to compare with the specific EOF instance to see if you reached end of file. You also need to actually return this value in custom code. Here I read from a TCP/IP socket into a buffer and get the number of bytes n read and any possible errors err. Because errors can be nested objects you should use errors.Is to compare rather than doing a err == io.EOF.

n, err := socket.Read(buffer)
if errors.Is(err, io.EOF) {
	break
}

You should familiarize yourself with all these error functions in the errors package:

func As(err error, target any) bool
func Is(err, target error) bool
func New(text string) error
func Unwrap(err error) error

Errors are wrapped using the fmt.Errorf function with the %w formatter.

newerr := fmt.Errorf("Stuff went wrong because %w", err)

Say several error objects have been nested and somewhere deep in the nesting there is a fs.PathError error you want to get hold of to obtain the Path which caused the error. You can do that with the As method:

var perr *fs.PathError
if errors.As(err, &perr) {
	fmt.Println(perr.Path)
}

Usually, you can return error objects by using fmt.Errorf but occasionally you need to create custom error objects with special fields. For an object to match the error interface, it just needs to implement the Error() method returning a string describing the error.

type MyError struct {
	When time.Time
	What string
}

func (e MyError) Error() string {
	return fmt.Sprintf("%v: %v", e.When, e.What)
}

Enums in Go

Much like errors, enums is pretty simple in Go. In fact, there isn’t really support for a special enum type. Instead, what you need to work with is a set of common practices and conventions around constants. Here is an example of defining an “enum” for some objects I made simulating Unix commands which had to return a result after they had executed.

//go:generate stringer -type=ExitCode

type ExitCode int

const (
	Ok      ExitCode = iota // normal result
	Failure                 // Command could not complete task
	Quit                    // request to quit shell
)

Notice how OK, Failure and Quit are not defined as simple integer values but a custom ExitCode type. Why do that? That way, it is easier to avoid accidentally passing a regular integer outside of the range. With iota these constants get the values 0, 1 and 2. We don't want other values than that.

It is not a foolproof system, as somebody could just write ExitCode(42) and pass it as an argument. Personally, I don't see that as a problem. The point is to not accidentally pass an invalid value.

Satisfy Stringer Interface

It is very common to want a text string version of your enum values. You could add the String() method to the Exit code and thus satisfy the Stringer interface.

type Stringer interface {
	String() string
}

A more convenient approach is to use the stringer command bundled with Go to automatically generate it. A simple approach is:

❯ stringer -type=ExitCode

It will look in the current directory to find a file defining the ExitCode type and look at the constants associated with it to add a String() method. It gets placed in the exitcode_string.go file. So just a lowercase version of the name with _string.go appended.

The command doesn’t always work. I had some issues with my code. It helps to put the enum in a separate file and specify that filename explicitly.

❯ stringer -type=ExitCode
stringer: internal error: package "bytes" without types was imported from "github.com/ordovician/mainframe"

# Fix error by specifying source code file directly
❯ stringer -type=ExitCode exitcode.go

Calling stringer for every enum type can get overwhelming to keep track of. We could of course make something akin to a Makefile, but the Go philosophy has been to try to avoid the usage of complex build scripts. The Go solution is to use the go generate command line tool. You put the command you want to run inside your source code files. That is why you see our enum definition start with:

//go:generate stringer -type=ExitCode
type ExitCode int

The Go convention is that comments starting with things like go: and json: can be read by a variety of Go tools. You see that with JSON marshaling and unmarshaling.

type User struct {
    Name string `json:"full_name"`
    Age int `json:"age,omitempty"`
    Active bool `json:"-"`
    lastLoginAt string
}

If you run go generate from the command line, it will visit all Go source code files and run the commands following //go:generate. That way we can keep building instructions with the code affected and avoid complex build systems which can become disconnected from the source code it is meant to build. You can move an individual source code file to another project, and it will carry with it instructions for how to build related artifacts it needs.

Using Non-Integer Enums

Your enums don’t have to be numbers. They can be anything. Here I am using strings. That can be useful when they represent stuff which is string input from users or files.

type Encoding string

const (
	PEM    Encoding = "pem"
	Hex    Encoding = "hex"
	Base32 Encoding = "base32"
	Base64 Encoding = "base64"
)

Of course, you will need to verify that what you have read is in a valid range.

Embedding data files in Go binaries

To make self-contained binaries, you can easily move around, it is useful to bundle data with them. Often that has meant hardcoding it into source code, but that can get messy, ugly and impractical.

import "embed"

//go:embed data
var storage embed.FS

This code allows me to read files from a directory in my package source code, which looks like this:

data/
├── agents.aes
└── launchcodes.txt

You can use the storage variable to access the embedded data as if it were regular files.

file, _ := storage.Open("data/launchcodes.txt")
text, _ := io.ReadAll(file)

In files where we embed a file as a string, we need a twist on how we import embeded because if an imported package isn't referenced in the code, the Go compiler complain. That it is referenced in the comment field does not get picked up by the compiler.

import _ "embed"

//go:embed key.txt
var cryptKey string

The dash _ is just a way to get the Go compiler to shut up.

Summary

Let me summarize what I have covered here in a more compact form for easier reference:

• Set GOPATH to ~/.go to keep a hidden directory for installed third-party packages and tools
• Keep your own projects in say ~/Dev/go
• Include $GOPATH/bin in your PATH so Go command line tools can be run easily.
• go mod init github.com/user/package - Initialize a Go module
• go mod tidy - Add and remove dependencies based on what packages your source code uses
• go build github.com/user/package/subpackage - Build a package or go build ./subpackage
• go test -v -run TestFoobar ./subpackage - Run specific tests TestFoobar written in sub package subpackage
• go install github.com/user/tool@latest - Install a tool or command written in Go
• io.Discard - Discard to throw away data written to a writer object. Useful when disabling loggers.
• //go:generate stringer -type=EnumName - To make an enum implement the Stringer interface
• //go:embed fileOrDir - Embed a file or whole directory in binary file

Next time I will probably try to cover the parts of Go generics which are easy to forget or mixup. Many people know a lot of these features but need a quick reminder.