Bug #15209
closedArvados Python packages have corresponding dependencies pinned
Added by Lucas Di Pentima over 5 years ago. Updated about 5 years ago.
Description
This is the last bulletpoint on #15080.
For example arvados-cwl-runner
1.3.3 package should ask for arvados-python-client
1.3.3
Related issues
Updated by Lucas Di Pentima over 5 years ago
- Related to Bug #15080: Installing arvados-cwl-runner==1.3.1 from PyPi is broken added
Updated by Lucas Di Pentima over 5 years ago
- Assigned To set to Lucas Di Pentima
- Target version set to 2019-05-22 Sprint
Updated by Lucas Di Pentima over 5 years ago
- Status changed from New to In Progress
- Start date set to 05/17/2019
Updated by Lucas Di Pentima over 5 years ago
- Status changed from In Progress to New
Updated by Lucas Di Pentima over 5 years ago
- Target version changed from 2019-05-22 Sprint to 2019-06-05 Sprint
Updated by Lucas Di Pentima about 5 years ago
- Target version changed from 2019-06-05 Sprint to 2019-06-19 Sprint
Updated by Lucas Di Pentima about 5 years ago
- Target version changed from 2019-06-19 Sprint to 2019-07-03 Sprint
Updated by Lucas Di Pentima about 5 years ago
- Status changed from New to In Progress
Updated by Lucas Di Pentima about 5 years ago
I have been trying different approaches and am somewhat blocked.
For what I've seen, there're 2 main cases when building packages:
1. New version release: the package version is provided by using an env variable and it's the same on all packages.
2. New dev package: the package version is built from the last tag (aka: release version) plus a timestamp. This timestamp may or may not be different across packages for every single commit.
The issue with changing timestamps is that I haven't found a clean way to get (for example) the arvados-python-client
version from the arvados-cwl-runner
package build process. I think we are using different timestamps to avoid creating/uploading new dev packages when components don't change from one commit to another.
Talking to Fernando about this, I learned that we're no longer uploading dev packages to PyPI, so maybe it won't be a big issue to start using the same timestamp on every component, so that every package build process knows that its own version matches the other packages versions.
If we should keep creating different timestamps for every dev package, maybe I should change the entire approach and have a completely external process that gets all the versions creating some kind of queryable common index?
Updated by Peter Amstutz about 5 years ago
Lucas Di Pentima wrote:
I have been trying different approaches and am somewhat blocked.
For what I've seen, there're 2 main cases when building packages:
1. New version release: the package version is provided by using an env variable and it's the same on all packages.
2. New dev package: the package version is built from the last tag (aka: release version) plus a timestamp. This timestamp may or may not be different across packages for every single commit.The issue with changing timestamps is that I haven't found a clean way to get (for example) the
arvados-python-client
version from thearvados-cwl-runner
package build process. I think we are using different timestamps to avoid creating/uploading new dev packages when components don't change from one commit to another.Talking to Fernando about this, I learned that we're no longer uploading dev packages to PyPI, so maybe it won't be a big issue to start using the same timestamp on every component, so that every package build process knows that its own version matches the other packages versions.
If we should keep creating different timestamps for every dev package, maybe I should change the entire approach and have a completely external process that gets all the versions creating some kind of queryable common index?
The most important is (1) so that installing from PyPi doesn't fetch incorrect dependencies (e.g. installing arvados-cwl-runner 1.3 doesn't fetch arvados-python-client 1.4)
For (2) perhaps for a-c-r or arvados-fuse version "X" the dependency is on "arvados-python-client is <= X" ? Then the default behavior is to get the latest that is equal or lesser than the version of the package being built. (Assuming PyPi doesn't complain if version X doesn't exist).
Updated by Lucas Di Pentima about 5 years ago
Updates at 3bc30ad82 - branch 15209-python-arv-deps-pinned
Ok, so after some bash script fiddling I think I got this working. Thanks Peter for your suggestions, they worked perfectly.
The python-arvados-client
sdist package is built even if --only-build
ask for another package, so it's available when other dependant packages are installed on their virtualenv. Pass -f <pysdk_sdist_path>
to pip so that it doesn't try to download arvados-python-client
from PyPI's index.
Updated by Lucas Di Pentima about 5 years ago
For testing, I ran:
$ WORKSPACE=$(pwd) ./build/run-build-packages-one-target.sh --test-packages --build-version "1.4.1-1"
Also, installed arvados-cwl-runner
's deb package and ran some example cwl workflow against 4xphq with both the fake 1.4.1 release version and also the dev version. In both cases, versions were correctly reported, for example:
lucas@veritas:~/cwl-examples$ arvados-cwl-runner --submit 1st-tool.cwl echo-job.yml INFO /usr/bin/arvados-cwl-runner 1.4.1, arvados-python-client 1.4.1, cwltool 1.0.20190607183319 INFO Resolved '1st-tool.cwl' to 'file:///home/lucas/cwl-examples/1st-tool.cwl' INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO Using collection cache size 256 MiB INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO [container 1st-tool.cwl] 4xphq-xvhdp-744yc2e7n7iofay state is Committed INFO [container 1st-tool.cwl] 4xphq-xvhdp-744yc2e7n7iofay is Final INFO Overall process status is success INFO Final output collection 282fbb8a925551343e98e762fa1ec1be+57 "Output of 1st-tool.cwl" (4xphq-4zz18-haoisak7o1k6kuj) {} INFO Final process status is success
Updated by Peter Amstutz about 5 years ago
Lucas Di Pentima wrote:
For testing, I ran:
[...]
Also, installed
arvados-cwl-runner
's deb package and ran some example cwl workflow against 4xphq with both the fake 1.4.1 release version and also the dev version. In both cases, versions were correctly reported, for example:[...]
This LGTM.
Updated by Lucas Di Pentima about 5 years ago
Fixed some silly mistakes at 4139c5dfd
Test run: https://ci.curoverse.com/job/developer-run-tests/1352/
Updated by Lucas Di Pentima about 5 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|bc14c62ad1528dbddc26781c5cea6a7968c93f2e.