Go dependency management for Python developers
How to manage dependencies on a Go project, compared to the Python ecosystem? How is the Go toolchain superior to Python in this regard?
I've written in the past about dependency management in Python with Poetry and pip-tools. Now that I've gained more experience with Go at OVHcloud (we're hiring), it's time to compare these two worlds!
📖 Terminology
Published units are called “packages” in Python, and “modules” in Go. You can find more information about Go modules in the documentation:
The central registry where to find Python packages is PyPI (Python Package Index). There's no actual equivalent in the Go world, but pkg.go.dev can be used to search for Go modules. More about this below.
🗺️ Working on a project with dependencies
Let's say you've been hired in a Go team (any resemblance to real persons, living or dead, is purely coincidental). Your first task consists of adding a feature to an existing project. You clone the repository and want to run tests to ensure that everything works properly before conducting your modifications.
How do you install dependencies? With a Python project, you'd create a virtual environment, activate it and install dependencies from a requirements.txt
file—or any other method depending on your team's tooling choices.
With Go, it's actually much quicker: just run go test ./...
in your repository. The toolchain will handle all the heavy lifting for you! The best part is that you practically never have to think about it. Checking out another revision from a colleague who has updated dependencies on their branch? No big deal! Run your usual go
commands to build, run or test your project. Tests won't fail because you forgot to pip install
.
Python | Go |
---|---|
clone | clone |
create virtualenv | run tests |
activate virtualenv | start coding |
install dependencies | iterate |
run tests | 🧘🏻♂️ |
start coding | 🧘🏻♂️ |
iterate | 🧘🏻♂️ |
go mod download
to download dependencies outside the context of another go
command.📈 Adding, updating or removing dependencies
Use go get
.
go get example.com/module
will add the latest version ofexample.com/module
to yourgo.mod
file if it isn't yet present, or update it to its latest version (@latest
is implied).go get example.com/module@v1.2.3
will set the version to the one you specified, updating or downgrading if needed.go get example.com/module@none
removes the dependency from thego.mod
file.
go get example.com/module@abc1234
is perfectly valid and will use the code as published in this specific commit. You can therefore use unreleased or even unmerged changes to test things out.It's good practice to run go mod tidy
regularly, most importantly after adding/updating/removing dependencies.
go mod tidy
ensures that thego.mod
file matches the source code in the module. It adds any missing module requirements necessary to build the current module’s packages and dependencies, and it removes requirements on modules that don’t provide any relevant packages. It also adds any missing entries togo.sum
and removes unnecessary entries.
— https://go.dev/ref/mod#go-mod-tidy (emphasis mine)
A corollary is that you can also remove a dependency by removing the code that imports it and running go mod tidy
.
To update all dependencies to their latest version, use go get -u ./...
.
More details in the go get documentation.
Compared to Python where the default tooling doesn't provide hash locking and requires manually running command and listing dependencies, Go shines by its simplicity.
🔍 Finding modules
There's no central registry where modules are published, unlike PyPI in the Python world or npmjs.com for Node.js. Go has the concept of module proxies instead. The default one is proxy.golang.org
. In a company setting, you'll likely have an internal proxy to configure with the GOPROXY
environment variable, in order to access private dependencies.
Public dependencies fetched at least once via proxy.golang.org
are available on pkg.go.dev, your one-stop shop for documentation.
Most likely, you'll find useful modules via a generic search engine query, which directs you to a git forge such as GitHub, GitLab or any other.
Go modules are usually go get
-able by their URL. For example, consider this popular logging library: sirupsen/logrus. You can use it with go get github.com/sirupsen/logrus
.
It's not always true, though. e.g., this testing library is hosted on GitHub, but is imported as gotest.tools
. Module authors tend to include this information in the project's README if necessary.
🗞️ Publishing a module
As we saw, there's no central registry where to publish your new shiny Go module. git push
! There's already a good and succinct documentation page:
When tagging, remember that Go is picky about version numbers. In particular, they must include a v
at the beginning. e.g., v1.2.3
is valid but 1.2.3
isn't.
v2 modules
When publishing a new major version, from v2 onward you will also need to add /vX
at the end for your module import path.
For example, let's consider you own a library hosted at example.com/lib
and currently at v1.2.3. To get it, you run go get example.com/lib@v1.2.3
. When you need to publish backward incompatible changes, you'll want to bump the major version number. However, you can't simply publish a new v2.0.0
tag. You'll also need to update the go.mod
to read module example.com/lib/v2
.
This also means that a new major version of a module is a whole new module as far as dependency resolution is concerned. You can thus import multiple major versions of a module in your project ; the new major version won't be offered when running go get -u
.
There is specific documentation for developing and publishing a new major version.
🏃🏻♂️ Going further
There are several more advanced topics that I don't discuss in this article, because I'm not very familiar with them:
- vendoring—including your actual dependencies in your repo, not just listing them in a
go.mod
file - workspaces - use unpublished modules locally
👨🏻🔬 Bonus - managing Go versions
The Go team makes tremendous efforts to be as backward and forward compatible as possible. They published two articles in August 2023 around the 1.21 release documenting their commitment:
Basically, a program built for Go 1.3 in 2014 should compile with Go 1.22 in 2024, and work as expected (or faster).
Furthermore, since Go 1.21, the Go toolchain will automatically download new versions when necessary. e.g., if you have Go 1.21 installed and checkout a module requiring Go 1.22, the toolchain will download Go 1.22 and run that instead. Conversely, if you have Go 1.22 installed and work on a Go 1.21 project, it will download and use Go 1.21 to run your commands.
So I'd argue that any version of Go >= 1.21 is the last one you need to manually install.
Now if you need to work on older projects:
- most of the time, you can upgrade your project to require a more recent Go version, and it will run fine;
- most of the time, you can use a more recent Go version than required by the project, and it will run fine;
- for the cases where 1 or 2 isn't possible, you can either:
- manually install alternative Go versions using your existing toolchain, which has the advantage of not requiring any additional software
- use a “toolchain manager” such as gvm (looks as if there are several versions available), asdf or mise.
I like mise
because it runs fast, doesn't depend on shims, and can set environment variables automatically on entering a directory.