Go dependencies Python OVHcloud

Go dependency management for Python developers

How to manage dependencies on a Go project, compared to the Python ecosystem? How is the Go toolchain superior to Python in this regard?

Gabriel Augendre

May 6, 2024 • 6 min read

I've written in the past about dependency management in Python with Poetry and pip-tools. Now that I've gained more experience with Go at OVHcloud (we're hiring), it's time to compare these two worlds!

💡

This article was written with modern Go versions in mind (1.22 at the time of writing). If you still use an older version which doesn't support Go modules, check the Bonus section first.

📖 Terminology

Published units are called “packages” in Python, and “modules” in Go. You can find more information about Go modules in the documentation:

The central registry where to find Python packages is PyPI (Python Package Index). There's no actual equivalent in the Go world, but pkg.go.dev can be used to search for Go modules. More about this below.

🗺️ Working on a project with dependencies

Let's say you've been hired in a Go team (any resemblance to real persons, living or dead, is purely coincidental). Your first task consists of adding a feature to an existing project. You clone the repository and want to run tests to ensure that everything works properly before conducting your modifications.

How do you install dependencies? With a Python project, you'd create a virtual environment, activate it and install dependencies from a requirements.txt file—or any other method depending on your team's tooling choices.

With Go, it's actually much quicker: just run go test ./... in your repository. The toolchain will handle all the heavy lifting for you! The best part is that you practically never have to think about it. Checking out another revision from a colleague who has updated dependencies on their branch? No big deal! Run your usual go commands to build, run or test your project. Tests won't fail because you forgot to pip install.

Python	Go
clone	clone
create virtualenv	run tests
activate virtualenv	start coding
install dependencies	iterate
run tests	🧘🏻‍♂️
start coding	🧘🏻‍♂️
iterate	🧘🏻‍♂️

💡

You can run go mod download to download dependencies outside the context of another go command.

📈 Adding, updating or removing dependencies

Use go get.

go get example.com/module will add the latest version of example.com/module to your go.mod file if it isn't yet present, or update it to its latest version (@latest is implied).
go get example.com/module@v1.2.3 will set the version to the one you specified, updating or downgrading if needed.
go get example.com/module@none removes the dependency from the go.mod file.

💡

A cool trick is that you can use any commit hash or branch name as the version, so go get example.com/module@abc1234 is perfectly valid and will use the code as published in this specific commit. You can therefore use unreleased or even unmerged changes to test things out.

It's good practice to run go mod tidy regularly, most importantly after adding/updating/removing dependencies.

go mod tidy ensures that the go.mod file matches the source code in the module. It adds any missing module requirements necessary to build the current module’s packages and dependencies, and it removes requirements on modules that don’t provide any relevant packages. It also adds any missing entries to go.sum and removes unnecessary entries.
— https://go.dev/ref/mod#go-mod-tidy (emphasis mine)

A corollary is that you can also remove a dependency by removing the code that imports it and running go mod tidy.

To update all dependencies to their latest version, use go get -u ./....

More details in the go get documentation.

Compared to Python where the default tooling doesn't provide hash locking and requires manually running command and listing dependencies, Go shines by its simplicity.

🔍 Finding modules

There's no central registry where modules are published, unlike PyPI in the Python world or npmjs.com for Node.js. Go has the concept of module proxies instead. The default one is proxy.golang.org. In a company setting, you'll likely have an internal proxy to configure with the GOPROXY environment variable, in order to access private dependencies.

Public dependencies fetched at least once via proxy.golang.org are available on pkg.go.dev, your one-stop shop for documentation.

Most likely, you'll find useful modules via a generic search engine query, which directs you to a git forge such as GitHub, GitLab or any other.

Go modules are usually go get-able by their URL. For example, consider this popular logging library: sirupsen/logrus. You can use it with go get github.com/sirupsen/logrus.

It's not always true, though. e.g., this testing library is hosted on GitHub, but is imported as gotest.tools. Module authors tend to include this information in the project's README if necessary.

🗞️ Publishing a module

As we saw, there's no central registry where to publish your new shiny Go module. git push! There's already a good and succinct documentation page:

When tagging, remember that Go is picky about version numbers. In particular, they must include a v at the beginning. e.g., v1.2.3 is valid but 1.2.3 isn't.

v2 modules

When publishing a new major version, from v2 onward you will also need to add /vX at the end for your module import path.

For example, let's consider you own a library hosted at example.com/lib and currently at v1.2.3. To get it, you run go get example.com/lib@v1.2.3. When you need to publish backward incompatible changes, you'll want to bump the major version number. However, you can't simply publish a new v2.0.0 tag. You'll also need to update the go.mod to read module example.com/lib/v2.

This also means that a new major version of a module is a whole new module as far as dependency resolution is concerned. You can thus import multiple major versions of a module in your project ; the new major version won't be offered when running go get -u.

There is specific documentation for developing and publishing a new major version.

🏃🏻‍♂️ Going further

There are several more advanced topics that I don't discuss in this article, because I'm not very familiar with them:

vendoring—including your actual dependencies in your repo, not just listing them in a go.mod file
workspaces - use unpublished modules locally

👨🏻‍🔬 Bonus - managing Go versions

The Go team makes tremendous efforts to be as backward and forward compatible as possible. They published two articles in August 2023 around the 1.21 release documenting their commitment:

Basically, a program built for Go 1.3 in 2014 should compile with Go 1.22 in 2024, and work as expected (or faster).

Furthermore, since Go 1.21, the Go toolchain will automatically download new versions when necessary. e.g., if you have Go 1.21 installed and checkout a module requiring Go 1.22, the toolchain will download Go 1.22 and run that instead. Conversely, if you have Go 1.22 installed and work on a Go 1.21 project, it will download and use Go 1.21 to run your commands.

So I'd argue that any version of Go >= 1.21 is the last one you need to manually install.

Now if you need to work on older projects:

most of the time, you can upgrade your project to require a more recent Go version, and it will run fine;
most of the time, you can use a more recent Go version than required by the project, and it will run fine;
for the cases where 1 or 2 isn't possible, you can either:
1. manually install alternative Go versions using your existing toolchain, which has the advantage of not requiring any additional software
2. use a “toolchain manager” such as gvm (looks as if there are several versions available), asdf or mise.

I like mise because it runs fast, doesn't depend on shims, and can set environment variables automatically on entering a directory.