Arvados

Proposed New Git Version Generation Scheme¶

Current Setup¶

Every package that wants to be able to have a Git-based version number has code like the following:

• Call git log -n1 with the right arguments to get the hash of the commit we want to use for versioning.
• Call version-at-commit.sh with that hash to turn it into a full version number string.
• Make the version number conform to language conventions.
• Save the version number in the appropriate place(s) for the build.

Challenges¶

• This code gets duplicated a lot: seven copies of arvados_version.py, three gemspec files, and probably more I don't know about.
• We have the usual code duplication problem of logic not being kept consistent. The initial 3.0.0 release of gems was buggy because gemfiles didn't have this logic right.
• Because this code generally has to be prepared to run outside of Git (e.g., from PyPI source), it's basically impossible to fail correctly: the code generally has to assume that if there's a problem generating a version from Git, then it just shouldn't, and use an existing static version instead. This makes it more difficult to notice and debug problems.

Proposal¶

A successor tool version-at-commit (probably written in a more ergonomic language than shell) takes over the entire process. It has the following responsibilities:

• It knows the source location of every package that wants to use a Git version.
• It knows each package's dependencies so it can find the right commit to use for versioning.
• It knows the language version conventions of each package.
• It knows how to write a static file inside that package's source tree to define string constants with all the necessary version information.

As an illustrative example, let's say we want to generate a new version number for services/fuse. The new versioning tool gets the Git commit hash by calling git log -n1 --first-parent -- build/version-at-commit.sh sdk/python services/fuse (note it knows about the sdk/python dependency), and then writes services/fuse/arvados_fuse/_version.py with the following:

__version__ = '3.1.0.dev20241209170209'
commit = '05e1189fb680150c7737fe957b43b314e48daeb1'
timestamp = '2024-12-09T17:02:09Z'
interdependency = '~=3.1.0.dev0'

Now services/fuse/setup.py just has to say:

version_info = runpy.run_path('arvados_fuse/_version.py')
setup(name='arvados_fuse',
      version=version_info['__version__'],
      install_requires=[
       f"arvados{version_info['interdependency']}",
        ...
      ],
      ...,
)

Details¶

version-at-commit still respects ARVADOS_BUILDING_VERSION. If that's set, it uses that directly for the version number, and generates an interdependency string to match (e.g., '==3.0.0').

Exactly where the tool writes version information, and how the package's build tool reads it, can vary by language. The goal should be that this tool just writes static strings in the source language's syntax, and that's simple enough that the build system can source-level include it, or whatever's easy. The tool should not try to edit existing files, just write new ones from scratch. These files should be gitignored (just like they are now) and do not need to be committed.

The introduction of this tool means any time you want to do any kind of build from Git, you must first run this tool before any other build steps. However, this doesn't really add any process overhead to our standard processes. Instead, it just means that our build orchestration scripts like run-tests.sh, run-build-packages.sh, etc. need to call this tool early on.

Benefits¶

It solves all the Challenges.

The package-level logic becomes much simpler and unconditional whether it's being built from Git, a source package, etc. This greatly reduces the chances that bugs arise in the less common code paths (e.g., doing a release build instead of a development build).

The change is transparent to most users most of the time. The only time to think about it is if you're developing other build tooling (e.g., editing run-tests.sh, adding a new build script, etc.). It makes no extra work when developers write branches or prepare a release. It does not require dedicated Git commits.

It reduces the implicit dependencies of source packages. Right now we have a handful of test Docker images that have to have Git installed just so the source code can generate a Git version. Now instead the version files can be generated before the Docker build/start, and then the Docker image only needs the language's standard build tooling to read the static files.

The approach doesn't lock us in to any particular build system. e.g., we can switch our Python packages to pyproject.toml in the near future, and version-at-commit is not deeply intertwined in the mechanics of old setuptools. Build changes like this might require us to write version information in a different file or syntax, but it's loose coupling, not deep nesting.

This is great. I like it. Let's do it.

I'm sorry that it took a month for me to get around to reviewing it.

At engineering meeting we talked about:

• I would like to write this in a declarative way, where there is a non-code configuration file in a standard format that describes each "package" in the Arvados monorepo, with information about what build system it uses, where to write version information, what other packages it depends on, etc.
• Once we have that, it would be nice if the tool could not just write version information but do source builds, run tests, etc. run-tests.sh has a lot of this information baked into its logic. Moving it to a declarative system would be nice.
• Having the tool be in Python with minimal dependencies seems like the lowest common denominator we can agree upon for bootstrapping.

A couple things shake out of this.

One, I think the configuration format should be TOML. tomllib was added to the Python stdlib as of 3.11, so as of Debian 12 it requires nothing extra. Debian 11 packages python3-toml which provides a compatible interface, so it's easy to get without setting up a virtualenv or other bootstrapping. If we accept this bit of temporary transitional pain, we get a configuration file format with comments and strong types including arrays. Our other options are:

• INI format parsed by configparser, which is human-friendly and in every Python stdlib but all the values are string-typed so we have to write more parsing code ourselves.
• JSON parsed by json which lacks comments and is kind of annoying for humans to write.
• YAML parsed by some non-standard library which adds a dependency and has several footguns.

Since the TOML format is strictly better than INI and the downsides compared to INI disappear over time I think it's the best call overall.

Two, the interface should probably be: command SUBCOMMAND [OPTIONS ...] [TARGET ...]. This gives us room to grow to support other operations in the future. Also, subcommands can be added over a series of tickets. The first subcommand can be write-version to fulfill this ticket.

Three, the implementation should probably be in a module directory with a __main__.py so we have room to grow (and write tests!). We can write a wrapper command.py that provides a thin wrapper for runpy.run_module() if that's helpful. Either way it still should not require pre-installation to use.