Feature #10812
closed
[CWL] Upload tool dependencies to separate collections
Added by Peter Amstutz almost 8 years ago.
Updated almost 8 years ago.
Description
Currently, all the scripts used in a CWL run are uploaded to a single collection. That means a change in any script will result in changing the collection PDH for all scripts, which invalidates reuse.
Change this behavior to upload the dependencies of each tool to a separate collection, so that a change one tool's scripts does not change the collection PDH of unrelated scripts.
- Description updated (diff)
- Assigned To set to Peter Amstutz
- Target version changed from Arvados Future Sprints to 2017-01-18 sprint
The branch 10812-similar-collections-check lgtm @ a6afada6
- Status changed from New to In Progress
- Target version changed from 2017-01-18 sprint to 2017-02-01 sprint
- Story points changed from 0.5 to 1.0
- in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user
- In init.py, please undo this indentation update
@@ -177,7 +177,7 @@ class ArvCwlRunner(object):
for p in proc_states["items"]:
self.on_message({
- "object_uuid": p["uuid"],
+ "object_uuid": p["uuid"],
"event_type": "update",
- In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?
- In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”
- In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?
- One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.
Radhika Chippada wrote:
- in init.py, --submit-runner-image: wondering if --submit-job-image rather than introducing yet another word to the end user
I named it that way because there is a similar option --submit-runner-ram.
This reminds me that I should probably add a test, too.
- In init.py, please undo this indentation update
[...]
Fixed.
- In done.py : Can “logtail = []” in the method logtail be renamed so that it is not confusing?
You're right. Renamed to "loglines".
- In done.py : extra set of parentheses in “logt.append((g.group(1), g.group(2)))”
No, the extra set of parenthesis make it a Python tuple. So it is not appending two items, it is appending one item which is a tuple.
- In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?
Yes, will do.
- One sdk/cwl test was failing for me yesterday with run-tests, but not anymore. Please make sure they are all passing after you merge all the three branches.
Yes, I did fix one failing test since last night when I asked you to review.
In fsaccess.py “record = self.api_client.workflows().get(uuid=url[6:]).execute()” : do we want to specify num_retries?
Fixed in 10812-cwl-separate-dependencies dc17bad908428e268b8ac1cab966902ce4c67d03
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by Tom Morris 
Updated by Radhika Chippada