Bug #21273
closed
Package builds on Py3.6 distros fail - cannot install setuptools>=60 but we're requesting >=66
Description
Problem: 5e2a9f3aeb1e72b9776f3a9974f8f4bffde83f8a from #21184 changed run-tests.sh to try to install setuptools>=66. However, setuptools 59 is the last release supporting Python 3.6, which is what Ubuntu 18.04 and Rocky 8 have. Those build-packages jobs fail like:
Package keep-exercise_2.8.0~dev20231207152146-1_amd64.deb not found, building Building deb (amd64) package for keep-exercise from tools/keep-exercise Package libpam-arvados-go_2.8.0~dev20231207152146-1_amd64.deb not found, building DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. Could not find a version that satisfies the requirement setuptools>=66 (from versions: 0.6b1, 0.6b2, 0.6b3, 0.6b4, 0.6rc1, 0.6rc2, 0.6rc3, 0.6rc4, 0.6rc5, 0.6rc6, 0.6rc7, 0.6rc8, 0.6rc9, 0.6rc10, 0.6rc11, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.7.8, 0.8, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 1.0, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.2, 1.3, 1.3.1, 1.3.2, 1.4, 1.4.1, 1.4.2, 2.0, 2.0.1, 2.0.2, 2.1, 2.1.1, 2.1.2, 2.2, 3.0, 3.0.1, 3.0.2, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.5, 3.5.1, 3.5.2, 3.6, 3.7, 3.7.1, 3.8, 3.8.1, 4.0, 4.0.1, 5.0, 5.0.1, 5.0.2, 5.1, 5.2, 5.3, 5.4, 5.4.1, 5.4.2, 5.5, 5.5.1, 5.6, 5.7, 5.8, 6.0.1, 6.0.2, 6.1, 7.0, 8.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.1, 8.2, 8.2.1, 8.3, 9.0, 9.0.1, 9.1, 10.0, 10.0.1, 10.1, 10.2, 10.2.1, 11.0, 11.1, 11.2, 11.3, 11.3.1, 12.0, 12.0.1, 12.0.2, 12.0.3, 12.0.4, 12.0.5, 12.1, 12.2, 12.3, 12.4, 13.0.1, 13.0.2, 14.0, 14.1, 14.1.1, 14.2, 14.3, 14.3.1, 15.0, 15.1, 15.2, 16.0, 17.0, 17.1, 17.1.1, 18.0, 18.0.1, 18.1, 18.2, 18.3, 18.3.1, 18.3.2, 18.4, 18.5, 18.6, 18.6.1, 18.7, 18.7.1, 18.8, 18.8.1, 19.0, 19.1, 19.1.1, 19.2, 19.3, 19.4, 19.4.1, 19.5, 19.6, 19.6.1, 19.6.2, 19.7, 20.0, 20.1, 20.1.1, 20.2.2, 20.3, 20.3.1, 20.4, 20.6.6, 20.6.7, 20.6.8, 20.7.0, 20.8.0, 20.8.1, 20.9.0, 20.10.1, 21.0.0, 21.1.0, 21.2.0, 21.2.1, 21.2.2, 22.0.0, 22.0.1, 22.0.2, 22.0.4, 22.0.5, 23.0.0, 23.1.0, 23.2.0, 23.2.1, 24.0.0, 24.0.1, 24.0.2, 24.0.3, 24.1.0, 24.1.1, 24.2.0, 24.2.1, 24.3.0, 24.3.1, 25.0.0, 25.0.1, 25.0.2, 25.1.0, 25.1.1, 25.1.2, 25.1.3, 25.1.4, 25.1.5, 25.1.6, 25.2.0, 25.3.0, 25.4.0, 26.0.0, 26.1.0, 26.1.1, 27.0.0, 27.1.0, 27.1.2, 27.2.0, 27.3.0, 27.3.1, 28.0.0, 28.1.0, 28.2.0, 28.3.0, 28.4.0, 28.5.0, 28.6.0, 28.6.1, 28.7.0, 28.7.1, 28.8.0, 28.8.1, 29.0.0, 29.0.1, 30.0.0, 30.1.0, 30.2.0, 30.2.1, 30.3.0, 30.4.0, 31.0.0, 31.0.1, 32.0.0, 32.1.0, 32.1.1, 32.1.2, 32.1.3, 32.2.0, 32.3.0, 32.3.1, 33.1.0, 33.1.1, 34.0.0, 34.0.1, 34.0.2, 34.0.3, 34.1.0, 34.1.1, 34.2.0, 34.3.0, 34.3.1, 34.3.2, 34.3.3, 34.4.0, 34.4.1, 35.0.0, 35.0.1, 35.0.2, 36.0.1, 36.1.0, 36.1.1, 36.2.0, 36.2.1, 36.2.2, 36.2.3, 36.2.4, 36.2.5, 36.2.6, 36.2.7, 36.3.0, 36.4.0, 36.5.0, 36.6.0, 36.6.1, 36.7.0, 36.7.1, 36.7.2, 36.8.0, 37.0.0, 38.0.0, 38.1.0, 38.2.0, 38.2.1, 38.2.3, 38.2.4, 38.2.5, 38.3.0, 38.4.0, 38.4.1, 38.5.0, 38.5.1, 38.5.2, 38.6.0, 38.6.1, 38.7.0, 39.0.0, 39.0.1, 39.1.0, 39.2.0, 40.0.0, 40.1.0, 40.1.1, 40.2.0, 40.3.0, 40.4.0, 40.4.1, 40.4.2, 40.4.3, 40.5.0, 40.6.0, 40.6.1, 40.6.2, 40.6.3, 40.7.0, 40.7.1, 40.7.2, 40.7.3, 40.8.0, 40.9.0, 41.0.0, 41.0.1, 41.1.0, 41.2.0, 41.3.0, 41.4.0, 41.5.0, 41.5.1, 41.6.0, 42.0.0, 42.0.1, 42.0.2, 43.0.0, 44.0.0, 44.1.0, 44.1.1, 45.0.0, 45.1.0, 45.2.0, 45.3.0, 46.0.0, 46.1.0, 46.1.1, 46.1.2, 46.1.3, 46.2.0, 46.3.0, 46.3.1, 46.4.0, 47.0.0, 47.1.0, 47.1.1, 47.2.0, 47.3.0, 47.3.1, 47.3.2, 48.0.0, 49.0.0, 49.0.1, 49.1.0, 49.1.1, 49.1.2, 49.1.3, 49.2.0, 49.2.1, 49.3.0, 49.3.1, 49.3.2, 49.4.0, 49.5.0, 49.6.0, 50.0.0, 50.0.1, 50.0.2, 50.0.3, 50.1.0, 50.2.0, 50.3.0, 50.3.1, 50.3.2, 51.0.0, 51.1.0, 51.1.0.post20201221, 51.1.1, 51.1.2, 51.2.0, 51.3.0, 51.3.1, 51.3.2, 51.3.3, 52.0.0, 53.0.0, 53.1.0, 54.0.0, 54.1.0, 54.1.1, 54.1.2, 54.1.3, 54.2.0, 56.0.0, 56.1.0, 56.2.0, 57.0.0, 57.1.0, 57.2.0, 57.3.0, 57.4.0, 57.5.0, 58.0.0, 58.0.1, 58.0.2, 58.0.3, 58.0.4, 58.1.0, 58.2.0, 58.3.0, 58.4.0, 58.5.0, 58.5.1, 58.5.2, 58.5.3, 59.0.1, 59.1.0, 59.1.1, 59.2.0, 59.3.0, 59.4.0, 59.5.0, 59.6.0) No matching distribution found for setuptools>=66 Error, unable to upgrade setuptools with pip3 install -q -U 'setuptools>=66' ERROR: build packages on arvados/build:ubuntu1804 failed with exit status 1
Solution: Our rocky8 build image should install the python39 appstream, and build against that.
Related issues
Updated by Brett Smith 5 months ago
- Related to Bug #21184: Fix build pipeline for debian 11 added
Updated by Brett Smith 5 months ago
- Related to Bug #21087: Python 3.7 deprecation added
Updated by Brett Smith 5 months ago
- Description updated (diff)
- Subject changed from Package builds on Py3.6 distros fail - cannot install setuptools>=60 but we're requesting >=68 to Package builds on Py3.6 distros fail - cannot install setuptools>=60 but we're requesting >=66
Updated by Brett Smith 5 months ago
In general we want to deal with this by dealing with #21087. Given that we're already planning on that, there's not much reason to tackle this separately.
I have removed build-packages-ubuntu1804 from build-packages-multijob, since that's ultimately where we were going to end up anyway.
build-packages-rocky8 is more involved, since its standard Python is too old but we want to support the distro generally and we still (AFAIK) haven't made a decision about how we're actually going to bridge that gap.
Updated by Brett Smith 4 months ago
- Description updated (diff)
Agreed at grooming we can update the rocky8 build to use the python39 appstream. That's the simplest available solution.
Updated by Peter Amstutz 4 months ago
- Target version changed from Development 2024-01-03 sprint to Development 2024-01-17 sprint
Updated by Brett Smith 4 months ago
21273-rocky8-python39 @ 0c179e6fd8d37c70227b0039375a1513582beeb1 - build-packages-rocky8: #277
This is built on top of the branch for #20846. I don't know if that was required per se, but definitely the branch is smarter about picking the right Python to use for building packages, and it probably didn't hurt. From there, it's literally just a matter of installing the right Python. Everything else adapts from there.
One thing is less than ideal: following Red Hat policy, the names of our packages should probably be python39-arvados-FOO, instead of python3-arvados-FOO like we're currently using. The current setup is technically fine, just maybe a little confusing for administrators. But given that we don't know of any current users using this package, I'm inclined to say that's more trouble than it's worth, and we should just document the issue. If Peter or Tom want to overrule me and say a metapackage is in scope for the ticket, that's fine, I'll do it.
- All agreed upon points are implemented / addressed.
- Yes
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
- Code is tested and passing, both automated and manual, what manual testing was done is described
- Yes, see above. The branch only changes build scripts (from its starting point) so there's nothing else to test.
- Documentation has been updated.
- Pending resolution of the naming question above. I should at least write an upgrade note to note we use the Python 3.9 package now. But it would be easier to write that note once I know whether we're sticking with the current naming, or if we're going to canonically name the packages with
python39.
- Pending resolution of the naming question above. I should at least write an upgrade note to note we use the Python 3.9 package now. But it would be easier to write that note once I know whether we're sticking with the current naming, or if we're going to canonically name the packages with
- Behaves appropriately at the intended scale (describe intended scale).
- N/A
- Considered backwards and forwards compatibility issues between client and server.
- This requires people using Arvados to switch their entire systems to Python 3.9, but I believe (hope?) we were on the same page about this when we chose this implementation plan.
- Follows our coding standards and GUI style guidelines.
- Yes
Updated by Brett Smith 4 months ago
I know this is unorthodox but I'm rolling the changes for this into the #20846 branch. Once all of those are done, the changes to get rocky8 working are trivial, so the overhead of a separate branch is just excessive. There is a dedicated commit for it at least, 0c179e6fd8d37c70227b0039375a1513582beeb1.
Updated by Brett Smith 4 months ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|b352c3862814fe0bdd2b5a40b1dc8171474dbb48.