Bug #15209

Arvados Python packages have corresponding dependencies pinned

Added by Lucas Di Pentima 3 months ago. Updated 24 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
05/17/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

This is the last bulletpoint on #15080.
For example arvados-cwl-runner 1.3.3 package should ask for arvados-python-client 1.3.3


Subtasks

Task #15323: Review 15209-python-arv-deps-pinnedResolvedPeter Amstutz


Related issues

Related to Arvados - Bug #15080: Installing arvados-cwl-runner==1.3.1 from PyPi is brokenResolved04/10/2019

Associated revisions

Revision bc14c62a
Added by Lucas Di Pentima 25 days ago

Merge branch '15209-python-arv-deps-pinned'
Closes #15209

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

History

#1 Updated by Lucas Di Pentima 3 months ago

  • Related to Bug #15080: Installing arvados-cwl-runner==1.3.1 from PyPi is broken added

#2 Updated by Lucas Di Pentima 3 months ago

  • Assigned To set to Lucas Di Pentima
  • Target version set to 2019-05-22 Sprint

#3 Updated by Lucas Di Pentima 2 months ago

  • Start date set to 05/17/2019
  • Status changed from New to In Progress

#4 Updated by Lucas Di Pentima 2 months ago

  • Status changed from In Progress to New

#5 Updated by Lucas Di Pentima 2 months ago

  • Target version changed from 2019-05-22 Sprint to 2019-06-05 Sprint

#6 Updated by Lucas Di Pentima about 2 months ago

  • Target version changed from 2019-06-05 Sprint to 2019-06-19 Sprint

#7 Updated by Lucas Di Pentima about 2 months ago

  • Tracker changed from Task to Bug

#8 Updated by Ward Vandewege about 1 month ago

  • Release set to 22

#9 Updated by Lucas Di Pentima about 1 month ago

  • Target version changed from 2019-06-19 Sprint to 2019-07-03 Sprint

#10 Updated by Lucas Di Pentima 29 days ago

  • Status changed from New to In Progress

#11 Updated by Lucas Di Pentima 27 days ago

I have been trying different approaches and am somewhat blocked.

For what I've seen, there're 2 main cases when building packages:

1. New version release: the package version is provided by using an env variable and it's the same on all packages.
2. New dev package: the package version is built from the last tag (aka: release version) plus a timestamp. This timestamp may or may not be different across packages for every single commit.

The issue with changing timestamps is that I haven't found a clean way to get (for example) the arvados-python-client version from the arvados-cwl-runner package build process. I think we are using different timestamps to avoid creating/uploading new dev packages when components don't change from one commit to another.

Talking to Fernando about this, I learned that we're no longer uploading dev packages to PyPI, so maybe it won't be a big issue to start using the same timestamp on every component, so that every package build process knows that its own version matches the other packages versions.

If we should keep creating different timestamps for every dev package, maybe I should change the entire approach and have a completely external process that gets all the versions creating some kind of queryable common index?

#12 Updated by Peter Amstutz 27 days ago

Lucas Di Pentima wrote:

I have been trying different approaches and am somewhat blocked.

For what I've seen, there're 2 main cases when building packages:

1. New version release: the package version is provided by using an env variable and it's the same on all packages.
2. New dev package: the package version is built from the last tag (aka: release version) plus a timestamp. This timestamp may or may not be different across packages for every single commit.

The issue with changing timestamps is that I haven't found a clean way to get (for example) the arvados-python-client version from the arvados-cwl-runner package build process. I think we are using different timestamps to avoid creating/uploading new dev packages when components don't change from one commit to another.

Talking to Fernando about this, I learned that we're no longer uploading dev packages to PyPI, so maybe it won't be a big issue to start using the same timestamp on every component, so that every package build process knows that its own version matches the other packages versions.

If we should keep creating different timestamps for every dev package, maybe I should change the entire approach and have a completely external process that gets all the versions creating some kind of queryable common index?

The most important is (1) so that installing from PyPi doesn't fetch incorrect dependencies (e.g. installing arvados-cwl-runner 1.3 doesn't fetch arvados-python-client 1.4)

For (2) perhaps for a-c-r or arvados-fuse version "X" the dependency is on "arvados-python-client is <= X" ? Then the default behavior is to get the latest that is equal or lesser than the version of the package being built. (Assuming PyPi doesn't complain if version X doesn't exist).

#13 Updated by Lucas Di Pentima 25 days ago

Updates at 3bc30ad82 - branch 15209-python-arv-deps-pinned

Ok, so after some bash script fiddling I think I got this working. Thanks Peter for your suggestions, they worked perfectly.

The python-arvados-client sdist package is built even if --only-build ask for another package, so it's available when other dependant packages are installed on their virtualenv. Pass -f <pysdk_sdist_path> to pip so that it doesn't try to download arvados-python-client from PyPI's index.

#14 Updated by Lucas Di Pentima 25 days ago

For testing, I ran:

$ WORKSPACE=$(pwd) ./build/run-build-packages-one-target.sh --test-packages --build-version "1.4.1-1" 

Also, installed arvados-cwl-runner's deb package and ran some example cwl workflow against 4xphq with both the fake 1.4.1 release version and also the dev version. In both cases, versions were correctly reported, for example:

lucas@veritas:~/cwl-examples$ arvados-cwl-runner --submit 1st-tool.cwl echo-job.yml 
INFO /usr/bin/arvados-cwl-runner 1.4.1, arvados-python-client 1.4.1, cwltool 1.0.20190607183319
INFO Resolved '1st-tool.cwl' to 'file:///home/lucas/cwl-examples/1st-tool.cwl'
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using collection cache size 256 MiB
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO [container 1st-tool.cwl] 4xphq-xvhdp-744yc2e7n7iofay state is Committed
INFO [container 1st-tool.cwl] 4xphq-xvhdp-744yc2e7n7iofay is Final
INFO Overall process status is success
INFO Final output collection 282fbb8a925551343e98e762fa1ec1be+57 "Output of 1st-tool.cwl" (4xphq-4zz18-haoisak7o1k6kuj)
{}
INFO Final process status is success

#15 Updated by Peter Amstutz 25 days ago

Lucas Di Pentima wrote:

For testing, I ran:

[...]

Also, installed arvados-cwl-runner's deb package and ran some example cwl workflow against 4xphq with both the fake 1.4.1 release version and also the dev version. In both cases, versions were correctly reported, for example:

[...]

This LGTM.

#17 Updated by Lucas Di Pentima 25 days ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF