Arvados

Python 3.12 will not preinstall setuptools in new virtualenvs. This will break our current build processes in various places. Doing this ticket would be the best way to get ahead of things.

Brett Smith wrote:

• Some of the interfaces of setup.py are deprecated (I think we're already getting warnings about this), and this gives us a clear migration path off those too.

To put more color on this: all of the interfaces of setup.py are deprecated. As the top of that post says,

This does not mean that setuptools itself is deprecated, or that using setup.py to configure your package builds is going to be removed. The only thing you must stop doing is directly executing the setup.py file — instead delegate that to purpose-built or standards-based tools, preferably those that work with any build backend.

We have been seeing this deprecation start to cause issues in some of our dependencies, which makes me think this migration is becoming more urgent.

For us, the path I think we want to take is: wherever we run setup.py install, run pip instead. We've already had a couple commits along these lines: see 20432a4533136a5ab9fa52c2e2ec2d90a855ecfb and upcoming d1e529905d4820ce450dc430139cccda83fefc72.

Wherever we run setup.py build, that needs to be updated to something like python -m build using the build module. There are other alternatives, this is just the simplest one I know about. However, before we can do that, we need to migrate to using at least pyproject.toml as specified in this ticket. Migrating setup.py build calls can either be part of this ticket, or a separate follow-up.

All our Python tools come with an identical arvados_version.py that knows how to generate the version number, interdependencies, and a _version.py submodule with this information. This is automatically imported and run from setup.py.

As we incrementally remove setup.py invocations, we're running into situations where tools do not need to run setup.py directly and therefore can run on modules without _version.py (which fails):

• pip install SOURCEDIR first copies the source to an isolated directory. This promotes build isolation but means that when arvados_version.py is run from that directory, it can't find any version information to work with.
• pytest doesn't necessarily need to run setup.py to run tests.

Right now all the smarts you need are in build/run-tests.sh, but because that's a big script doing a lot it means it's not obvious to follow, and working on it can be error-prone if you don't deeply understand the order of steps and why they're required.

As we look at migrating to pyproject.toml, I think we need a fresh look at our options for dynamically generating this metadata, and try to do so in a way that more tools will pick up.