Arvados

Proposed New Git Version Generation Scheme¶

Current Setup¶

Every package that wants to be able to have a Git-based version number has code like the following:

• Call git log -n1 with the right arguments to get the hash of the commit we want to use for versioning.
• Call version-at-commit.sh with that hash to turn it into a full version number string.
• Make the version number conform to language conventions.
• Save the version number in the appropriate place(s) for the build.

Challenges¶

• This code gets duplicated a lot: seven copies of arvados_version.py, three gemspec files, and probably more I don't know about.
• We have the usual code duplication problem of logic not being kept consistent. The initial 3.0.0 release of gems was buggy because gemfiles didn't have this logic right.
• Because this code generally has to be prepared to run outside of Git (e.g., from PyPI source), it's basically impossible to fail correctly: the code generally has to assume that if there's a problem generating a version from Git, then it just shouldn't, and use an existing static version instead. This makes it more difficult to notice and debug problems.

Proposal¶

A successor tool version-at-commit (probably written in a more ergonomic language than shell) takes over the entire process. It has the following responsibilities:

• It knows the source location of every package that wants to use a Git version.
• It knows each package's dependencies so it can find the right commit to use for versioning.
• It knows the language version conventions of each package.
• It knows how to write a static file inside that package's source tree to define string constants with all the necessary version information.

As an illustrative example, let's say we want to generate a new version number for services/fuse. The new versioning tool gets the Git commit hash by calling git log -n1 --first-parent -- build/version-at-commit.sh sdk/python services/fuse (note it knows about the sdk/python dependency), and then writes services/fuse/arvados_fuse/_version.py with the following:

__version__ = '3.1.0.dev20241209170209'
commit = '05e1189fb680150c7737fe957b43b314e48daeb1'
timestamp = '2024-12-09T17:02:09Z'
interdependency = '~=3.1.0.dev0'

Now services/fuse/setup.py just has to say:

version_info = runpy.run_path('arvados_fuse/_version.py')
setup(name='arvados_fuse',
      version=version_info['__version__'],
      install_requires=[
       f"arvados{version_info['interdependency']}",
        ...
      ],
      ...,
)

Details¶

version-at-commit still respects ARVADOS_BUILDING_VERSION. If that's set, it uses that directly for the version number, and generates an interdependency string to match (e.g., '==3.0.0').

Exactly where the tool writes version information, and how the package's build tool reads it, can vary by language. The goal should be that this tool just writes static strings in the source language's syntax, and that's simple enough that the build system can source-level include it, or whatever's easy. The tool should not try to edit existing files, just write new ones from scratch. These files should be gitignored (just like they are now) and do not need to be committed.

The introduction of this tool means any time you want to do any kind of build from Git, you must first run this tool before any other build steps. However, this doesn't really add any process overhead to our standard processes. Instead, it just means that our build orchestration scripts like run-tests.sh, run-build-packages.sh, etc. need to call this tool early on.

Benefits¶

It solves all the Challenges.

The package-level logic becomes much simpler and unconditional whether it's being built from Git, a source package, etc. This greatly reduces the chances that bugs arise in the less common code paths (e.g., doing a release build instead of a development build).

The change is transparent to most users most of the time. The only time to think about it is if you're developing other build tooling (e.g., editing run-tests.sh, adding a new build script, etc.). It makes no extra work when developers write branches or prepare a release. It does not require dedicated Git commits.

It reduces the implicit dependencies of source packages. Right now we have a handful of test Docker images that have to have Git installed just so the source code can generate a Git version. Now instead the version files can be generated before the Docker build/start, and then the Docker image only needs the language's standard build tooling to read the static files.

The approach doesn't lock us in to any particular build system. e.g., we can switch our Python packages to pyproject.toml in the near future, and version-at-commit is not deeply intertwined in the mechanics of old setuptools. Build changes like this might require us to write version information in a different file or syntax, but it's loose coupling, not deep nesting.