Feature #10812: [CWL] Upload tool dependencies to separate collections - Arvados

Actions

Copy link

Feature #10812

closed

[CWL] Upload tool dependencies to separate collections

Added by Peter Amstutz almost 8 years ago. Updated almost 8 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Peter Amstutz

Category:

Target version:

2017-02-01 sprint

Story points:

1.0

Description

Currently, all the scripts used in a CWL run are uploaded to a single collection. That means a change in any script will result in changing the collection PDH for all scripts, which invalidates reuse.

Change this behavior to upload the dependencies of each tool to a separate collection, so that a change one tool's scripts does not change the collection PDH of unrelated scripts.

Subtasks 4 (0 open — 4 closed)

Actions

Copy link

Updated by Peter Amstutz almost 8 years ago

Description updated (diff)

Actions

Copy link

Updated by Tom Morris almost 8 years ago

Assigned To set to Peter Amstutz
Target version changed from Arvados Future Sprints to 2017-01-18 sprint

Actions

Copy link

Updated by Radhika Chippada almost 8 years ago

The branch 10812-similar-collections-check lgtm @ a6afada6

Actions

Copy link

Updated by Peter Amstutz almost 8 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Peter Amstutz almost 8 years ago

Target version changed from 2017-01-18 sprint to 2017-02-01 sprint

Actions

Copy link

Updated by Peter Amstutz almost 8 years ago

Story points changed from 0.5 to 1.0

Actions

Copy link

Updated by Radhika Chippada almost 8 years ago

in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user

In init.py, please undo this indentation update

@@ -177,7 +177,7 @@ class ArvCwlRunner(object): 
                 for p in proc_states["items"]:
                     self.on_message({
-                        "object_uuid": p["uuid"],
+                       "object_uuid": p["uuid"],
                         "event_type": "update",

In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?

In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”

In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?

One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.

Actions

Copy link

Updated by Peter Amstutz almost 8 years ago

Radhika Chippada wrote:

in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user

I named it that way because there is a similar option --submit-runner-ram.

This reminds me that I should probably add a test, too.

In init.py, please undo this indentation update
[...]

Fixed.

In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?

You're right. Renamed to "loglines".

In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”

No, the extra set of parenthesis make it a Python tuple. So it is not appending two items, it is appending one item which is a tuple.

In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?

Yes, will do.

One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.

Yes, I did fix one failing test since last night when I asked you to review.

Actions

Copy link