Idea #21453
closed
Install Python package virtualenvs under /usr/lib/PKGNAME
Description
Background: We used to ship Python packages that actually integrated with the distro packages, so we installed Python stuff under /usr/share/python3/dist/PKGNAME because that's where you put system-wide Python libraries.
When we switched to shipping an entire virtualenv, we didn't change this path. It doesn't hurt anything but it's kind of weird. So as of Arvados 2.7, we ship an entire virtualenv under /usr/share/python3/dist/PKGNAME.
As part of #20846, we updated the build process to consistently specify a minor version of Python, to avoid accidental breakage if Python got upgraded. As part of that change, the path we install under is now /usr/share/python3.NN/dist/PKGNAME. This is even weirder because /usr/share/python3.NN isn't really an established path for anything.
Since we're updating the path anyway, we might as well just make it /usr/lib/PKGNAME. This follows established convention better, it's easier to follow, and it's a very straightforward change.
This should be documented in the upgrade notes. If people have custom scripts that point to one of our shipped virtualenvs, they'll need to update those paths. This is already true after #20846, so making the path nicer makes the upgrade note nicer too.
Related issues
Updated by Brett Smith 10 months ago
- Related to Idea #20846: Support Ubuntu 22.04 LTS added
Updated by Peter Amstutz 10 months ago
Just to be pendantic, /usr/share is supposed to be "architecture independent" files, but I believe the virtualenv can include binary wheels and stuff that's architecture specific, so it might be better suited to go under /usr/lib/PKGNAME?
Updated by Brett Smith 10 months ago
- Description updated (diff)
- Subject changed from Install Python package virtualenvs under /usr/share/PKGNAME to Install Python package virtualenvs under /usr/lib/PKGNAME
Updated by Brett Smith 9 months ago
- Status changed from New to In Progress
21453-python-package-path @ 8c33c582e6fa415dbdca5204f5926fe09de34e3b - build-packages-debian12: #38 - Can't test all package builds because of #21501
- All agreed upon points are implemented / addressed.
- Yes
- Anything not implemented (discovered or discussed during work) has a follow-up story.
- N/A
- Code is tested and passing, both automated and manual, what manual testing was done is described
- In addition to the Jenkins build above, I also built and tested the package locally. I then extracted the contents and confirmed it had the desired layout, with correct shebang lines for scripts under
/usr/bin.
- In addition to the Jenkins build above, I also built and tested the package locally. I then extracted the contents and confirmed it had the desired layout, with correct shebang lines for scripts under
- Documentation has been updated.
- Yes
- Behaves appropriately at the intended scale (describe intended scale).
- No change in scale
- Considered backwards and forwards compatibility issues between client and server.
- As described in the story, #20846 already changed the location of the virtualenv. This story improves the location and adds documentation.
- Follows our coding standards and GUI style guidelines.
- Yes
Updated by Peter Amstutz 9 months ago
- Target version changed from Development 2024-02-14 sprint to Development 2024-02-28 sprint
Updated by Lucas Di Pentima 9 months ago
Even though the run-test pipeline succeeded (developer-run-tests: #4042 ), I think these files should be fixed to use the new path:
build/package-testing/test-package-python3-arvados-python-client.sh:L10docker/jobs/Dockerfile:L23sdk/cwl/tests/arvados-tests.sh:L27
The rest LGTM, thanks!
Updated by Brett Smith 9 months ago
Lucas Di Pentima wrote in #note-6:
Even though the run-test pipeline succeeded (developer-run-tests: #4042
), I think these files should be fixed to use the new path:
These are all good catches, thank you, and done. To explain:
build/package-testing/test-package-python3-arvados-python-client.sh:L10
This test ran as part of build-packages-debian12, but because of #21526, it tested an already-built package with the old path, not the newly built one. We would have to do something dramatic to get the tests to the test the new package as intended.
Now that I've fixed this path, it's possible that build-packages jobs are going to start failing, because they're going to run the new test code against an old package. If that happens, the quickest fix is to push some commit that affects sdk/python to force a new package build. I'm sure I can find some ancient Python 2 compatibility import to clean up to help that process along.
docker/jobs/Dockerfile:L23
This only runs as part of docker-jobs-image, and isn't tested in any way as far as I know. Because this job only takes one Git hash as a parameter, I don't know if it's even possible to run it in a way that builds from a development package.
sdk/cwl/tests/arvados-tests.sh:L27
These run as part of the arvados-cwl-conformance-tests Jenkins job, which is not part of either developer-run-tests or build-packages. It's also affected by the "no way to select packages installed by a Dockerfile" issue as arvados/jobs.
Updated by Brett Smith 9 months ago
- % Done changed from 0 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|f8351929a502d46a1f95a2b270466ffeb16bf307.