Idea #20311
openUpdate Python packages to build with PEP 517/518
Description
Assuming we want to continue to build with setuptools, this means converting our packages' current setup.py
to setup.cfg
, then adding a (boilerplate) setup.py
and pyproject.toml
to them.
Reasons to do this:
- Gives us the option to build packages using other systems in the future if we need or want that for any reason.
- In my experience, maintining
setup.cfg
is nicer thansetup.py
, since it's more declarative. - Some of the interfaces of
setup.py
are deprecated (I think we're already getting warnings about this), and this gives us a clear migration path off those too.
Background reading:
- PyPA documentation
- The PEPs
- Brett Cannon blogs: 1, 2
- Here's a commit of me converting a project at a previous job so you can see what it looks like in action
Related issues
Updated by Brett Smith about 1 year ago
- Related to Idea #20723: Stop running setup.py in our build+test infrastructure added
Updated by Brett Smith about 1 year ago
Python 3.12 will not preinstall setuptools in new virtualenvs. This will break our current build processes in various places. Doing this ticket would be the best way to get ahead of things.
Updated by Brett Smith 11 months ago
Brett Smith wrote:
- Some of the interfaces of
setup.py
are deprecated (I think we're already getting warnings about this), and this gives us a clear migration path off those too.
To put more color on this: all of the interfaces of setup.py
are deprecated. As the top of that post says,
This does not mean that setuptools itself is deprecated, or that using setup.py to configure your package builds is going to be removed. The only thing you must stop doing is directly executing the setup.py file — instead delegate that to purpose-built or standards-based tools, preferably those that work with any build backend.
We have been seeing this deprecation start to cause issues in some of our dependencies, which makes me think this migration is becoming more urgent.
For us, the path I think we want to take is: wherever we run setup.py install
, run pip
instead. We've already had a couple commits along these lines: see 20432a4533136a5ab9fa52c2e2ec2d90a855ecfb and upcoming d1e529905d4820ce450dc430139cccda83fefc72.
Wherever we run setup.py build
, that needs to be updated to something like python -m build
using the build module. There are other alternatives, this is just the simplest one I know about. However, before we can do that, we need to migrate to using at least pyproject.toml
as specified in this ticket. Migrating setup.py build
calls can either be part of this ticket, or a separate follow-up.
Updated by Peter Amstutz 5 months ago
- Target version changed from Future to Development 2024-05-08 sprint
Updated by Brett Smith 5 months ago
- Target version changed from Development 2024-05-08 sprint to Future
Updated by Peter Amstutz 5 months ago
- Related to Bug #21721: Review Python SDK dependencies added
Updated by Brett Smith 3 months ago
All our Python tools come with an identical arvados_version.py
that knows how to generate the version number, interdependencies, and a _version.py
submodule with this information. This is automatically imported and run from setup.py
.
As we incrementally remove setup.py
invocations, we're running into situations where tools do not need to run setup.py
directly and therefore can run on modules without _version.py
(which fails):
pip install SOURCEDIR
first copies the source to an isolated directory. This promotes build isolation but means that whenarvados_version.py
is run from that directory, it can't find any version information to work with.pytest
doesn't necessarily need to runsetup.py
to run tests.
Right now all the smarts you need are in build/run-tests.sh
, but because that's a big script doing a lot it means it's not obvious to follow, and working on it can be error-prone if you don't deeply understand the order of steps and why they're required.
As we look at migrating to pyproject.toml
, I think we need a fresh look at our options for dynamically generating this metadata, and try to do so in a way that more tools will pick up.