Feature #9766

[Crunch2] Submit to workflows table

Added by Peter Amstutz almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
09/01/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0
Release:
Release relationship:
Auto

Description

arvados-cwl-runner --create-workflow workflow.cwl [defaults.yml]
arvados-cwl-runner --update-workflow=uuid workflow.cwl [defaults.yml]

Similar to --create-template:

  1. Upload dependent files to keep
  2. Rewrite file locations to keep locations
  3. Pack dependent tools into single document & rewrite cross references
  4. Create workflow record with packed workflow object, label -> name, doc -> description

Subtasks

Task #9889: Review 9766-register-workflowResolvedRadhika Chippada

Associated revisions

Revision cae94f22
Added by Peter Amstutz over 3 years ago

Merge branch '9766-register-workflow' closes #9766

Revision cc8a6bcb (diff)
Added by Peter Amstutz over 3 years ago

Fix misake in arvados-cwl-runner upload_dependencies refactor. refs #9766

Revision bb216349 (diff)
Added by Peter Amstutz over 3 years ago

Fix arvados-cwl-runner tests, refs #9766

History

#1 Updated by Tom Morris almost 4 years ago

Create workflow object. Take all the dependent files and make sure that they're. If workflow has multiple tools, package them all up.

#2 Updated by Peter Amstutz almost 4 years ago

  • Description updated (diff)

#3 Updated by Peter Amstutz almost 4 years ago

  • Story points set to 2.0

#4 Updated by Peter Amstutz almost 4 years ago

  • Description updated (diff)

#5 Updated by Peter Amstutz almost 4 years ago

  • Description updated (diff)

#6 Updated by Peter Amstutz almost 4 years ago

  • Target version set to 2016-08-31 sprint

#7 Updated by Peter Amstutz almost 4 years ago

  • Status changed from New to In Progress
  • Assigned To set to Peter Amstutz

#8 Updated by Radhika Chippada almost 4 years ago

  • Target version changed from 2016-08-31 sprint to 2016-09-14 sprint

#9 Updated by Peter Amstutz almost 4 years ago

  • Story points changed from 2.0 to 1.0

#10 Updated by Radhika Chippada almost 4 years ago

  • exgroup.add_argument("--update-workflow", type=str, help="Update Arvados workflow." => Can we call it --update-workflow-uuid? Please expand the description to say uuid of the workflow specified.
  • main = [p for p in packed["$graph"] if p["id"] == "#main"][0] => Is it always guaranteed that $graph is present here? Do we need any error handling in make_workflow?
  • It appears that the workflow being used is surrounded in a ”$graph": []. How does this compare with what we have in our fixtures? Do we need a fixture with a format similar to what you have here and verify how running such a workflow using workbench would work and how the inputs in workbench are rendered in such a case? Also, the code and examples are using multiple steps and I am wondering if you checked how such a workflow will be rendered in workbench.
  • Do we need a test with invalid cwl?
  • Does it make sense to add a yaml file based test?

#11 Updated by Peter Amstutz over 3 years ago

Radhika Chippada wrote:

  • exgroup.add_argument("--update-workflow", type=str, help="Update Arvados workflow." => Can we call it --update-workflow-uuid? Please expand the description to say uuid of the workflow specified.

Tweaked description and set metavar="UUID".

  • main = [p for p in packed["$graph"] if p["id"] == "#main"][0] => Is it always guaranteed that $graph is present here?

Yes.

Do we need any error handling in make_workflow?

No, I don't think so. Exceptions are caught higher up.

  • It appears that the workflow being used is surrounded in a ”$graph": []. How does this compare with what we have in our fixtures? Do we need a fixture with a format similar to what you have here and verify how running such a workflow using workbench would work and how the inputs in workbench are rendered in such a case? Also, the code and examples are using multiple steps and I am wondering if you checked how such a workflow will be rendered in workbench.

I adjusted the "uncommitted" fixture to use the $graph/#main pattern form that

  • Do we need a test with invalid cwl?

arvados-cwl-runner validates the file before it uploads it. Teaching workbench how to validate CWL is out of the scope of this ticket.

  • Does it make sense to add a yaml file based test?

I changed it to dump yaml text instead of json, and updated the test.

#12 Updated by Radhika Chippada over 3 years ago

LGTM

#13 Updated by Peter Amstutz over 3 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:cae94f22b760c6c6899fc4d23db15d389535ff0a.

Also available in: Atom PDF