Project

General

Profile

Actions

Feature #10812

closed

[CWL] Upload tool dependencies to separate collections

Added by Peter Amstutz over 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
1.0

Description

Currently, all the scripts used in a CWL run are uploaded to a single collection. That means a change in any script will result in changing the collection PDH for all scripts, which invalidates reuse.

Change this behavior to upload the dependencies of each tool to a separate collection, so that a change one tool's scripts does not change the collection PDH of unrelated scripts.


Subtasks 4 (0 open4 closed)

Task #10938: Review 10812-submit-runner-imageResolvedRadhika Chippada01/12/2017Actions
Task #10889: Review 10812-cwl-separate-dependenciesResolvedRadhika Chippada01/12/2017Actions
Task #10939: Review 10812-log-mergeResolvedRadhika Chippada01/12/2017Actions
Task #10831: Review 10812-similar-collections-checkResolvedRadhika Chippada01/12/2017Actions
Actions #1

Updated by Peter Amstutz over 7 years ago

  • Description updated (diff)
Actions #2

Updated by Tom Morris over 7 years ago

  • Assigned To set to Peter Amstutz
  • Target version changed from Arvados Future Sprints to 2017-01-18 sprint
Actions #3

Updated by Radhika Chippada over 7 years ago

The branch 10812-similar-collections-check lgtm @ a6afada6

Actions #4

Updated by Peter Amstutz over 7 years ago

  • Status changed from New to In Progress
Actions #5

Updated by Peter Amstutz over 7 years ago

  • Target version changed from 2017-01-18 sprint to 2017-02-01 sprint
Actions #6

Updated by Peter Amstutz over 7 years ago

  • Story points changed from 0.5 to 1.0
Actions #7

Updated by Radhika Chippada over 7 years ago

  • in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user
  • In init.py, please undo this indentation update
    @@ -177,7 +177,7 @@ class ArvCwlRunner(object): 
                     for p in proc_states["items"]:
                         self.on_message({
    -                        "object_uuid": p["uuid"],
    +                       "object_uuid": p["uuid"],
                             "event_type": "update",
    
  • In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?
  • In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”
  • In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?
  • One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.
Actions #8

Updated by Peter Amstutz over 7 years ago

Radhika Chippada wrote:

  • in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user

I named it that way because there is a similar option --submit-runner-ram.

This reminds me that I should probably add a test, too.

  • In init.py, please undo this indentation update
    [...]

Fixed.

  • In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?

You're right. Renamed to "loglines".

  • In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”

No, the extra set of parenthesis make it a Python tuple. So it is not appending two items, it is appending one item which is a tuple.

  • In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?

Yes, will do.

  • One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.

Yes, I did fix one failing test since last night when I asked you to review.

Actions #9

Updated by Peter Amstutz over 7 years ago

Updated 10812-submit-runner-image with tests, now 1cdb5cbf83d34cb1a19c78478a7d5a28738e7bbf

Updated 10812-log-merge now e8521124b0b8e647c58e99905100171ea5e30abd

Updated 10812-cwl-separate-dependencies now 1bf2e116aae97f0a6a9ed0962cd74e092e3b4695 running tests https://ci.curoverse.com/job/developer-run-tests/145/

Actions #10

Updated by Peter Amstutz over 7 years ago

In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?

Fixed in 10812-cwl-separate-dependencies dc17bad908428e268b8ac1cab966902ce4c67d03

Actions #11

Updated by Peter Amstutz about 7 years ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF