Feature #10812

[CWL] Upload tool dependencies to separate collections

Added by Peter Amstutz over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
01/12/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

Currently, all the scripts used in a CWL run are uploaded to a single collection. That means a change in any script will result in changing the collection PDH for all scripts, which invalidates reuse.

Change this behavior to upload the dependencies of each tool to a separate collection, so that a change one tool's scripts does not change the collection PDH of unrelated scripts.


Subtasks

Task #10938: Review 10812-submit-runner-imageResolvedRadhika Chippada

Task #10889: Review 10812-cwl-separate-dependenciesResolvedRadhika Chippada

Task #10939: Review 10812-log-mergeResolvedRadhika Chippada

Task #10831: Review 10812-similar-collections-checkResolvedRadhika Chippada

Associated revisions

Revision 6951a378
Added by Peter Amstutz over 4 years ago

Merge branch '10812-similar-collections-check' refs #10812

Revision 86328020
Added by Peter Amstutz over 4 years ago

Merge branch '10812-cwl-package-update' refs #10812

Revision b6e15a58
Added by Peter Amstutz over 4 years ago

Merge branch '10812-submit-runner-image' refs #10812

Revision c40389e0
Added by Peter Amstutz over 4 years ago

Merge branch '10812-log-merge' refs #10812

Revision 39189b90
Added by Peter Amstutz over 4 years ago

Merge branch '10812-cwl-separate-dependencies' refs #10812

Revision a3ca6693
Added by Peter Amstutz over 4 years ago

Merge branch '10812-docker-upload-fix' refs #10812

Revision a7222f4d (diff)
Added by Peter Amstutz over 4 years ago

arv_docker_get_image should not modify dockerRequirement parameter. refs #10812

History

#1 Updated by Peter Amstutz over 4 years ago

  • Description updated (diff)

#2 Updated by Tom Morris over 4 years ago

  • Assigned To set to Peter Amstutz
  • Target version changed from Arvados Future Sprints to 2017-01-18 sprint

#3 Updated by Radhika Chippada over 4 years ago

The branch 10812-similar-collections-check lgtm @ a6afada6

#4 Updated by Peter Amstutz over 4 years ago

  • Status changed from New to In Progress

#5 Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2017-01-18 sprint to 2017-02-01 sprint

#6 Updated by Peter Amstutz over 4 years ago

  • Story points changed from 0.5 to 1.0

#7 Updated by Radhika Chippada over 4 years ago

  • in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user
  • In init.py, please undo this indentation update
    @@ -177,7 +177,7 @@ class ArvCwlRunner(object): 
                     for p in proc_states["items"]:
                         self.on_message({
    -                        "object_uuid": p["uuid"],
    +                       "object_uuid": p["uuid"],
                             "event_type": "update",
    
  • In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?
  • In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”
  • In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?
  • One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.

#8 Updated by Peter Amstutz over 4 years ago

Radhika Chippada wrote:

  • in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user

I named it that way because there is a similar option --submit-runner-ram.

This reminds me that I should probably add a test, too.

  • In init.py, please undo this indentation update
    [...]

Fixed.

  • In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?

You're right. Renamed to "loglines".

  • In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”

No, the extra set of parenthesis make it a Python tuple. So it is not appending two items, it is appending one item which is a tuple.

  • In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?

Yes, will do.

  • One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.

Yes, I did fix one failing test since last night when I asked you to review.

#9 Updated by Peter Amstutz over 4 years ago

Updated 10812-submit-runner-image with tests, now 1cdb5cbf83d34cb1a19c78478a7d5a28738e7bbf

Updated 10812-log-merge now e8521124b0b8e647c58e99905100171ea5e30abd

Updated 10812-cwl-separate-dependencies now 1bf2e116aae97f0a6a9ed0962cd74e092e3b4695 running tests https://ci.curoverse.com/job/developer-run-tests/145/

#10 Updated by Peter Amstutz over 4 years ago

In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?

Fixed in 10812-cwl-separate-dependencies dc17bad908428e268b8ac1cab966902ce4c67d03

#11 Updated by Peter Amstutz over 4 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF