Project

General

Profile

Actions

Idea #15535

closed

[CWL] Run from original CWL, not packed

Added by Peter Amstutz over 4 years ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Story points:
3.0

Description

arvados-cwl-runner, on submitting a workflow, uses "pack" to create a single-stream document.

This is because the "workflow" record used to display workflows on workbench only stores a single raw text field, into which the multi-document CWL file has to be stuffed. The rationale for having a workflow record be a text field and not a PDH or git hash to avoid requiring workbench be able to fetch a collection / git repo to display a workflow. Although this isn't a limitation when submitting from the command line, this also uses the "pack" function to minimize having multiple code paths.

Unfortunately the packed version often bares little resemblance to user's original document. It would be better to execute the original document.

Proposal:

At CLI: Upload original workflow files & dependencies to a collection, preserving original filesystem structure. Submit a container request that mounts the collection and runs the workflow.

At workbench: to register workflow record, create a wrapper workflow that has the same input/output interface as the workflow, with a single step with a run line like:

run: keep:pdh/workflow

To submit the workflow, workbench introspects the step and sets up the correct collection mount.

To display cwl-svg of a workflow, workbench2 needs to be able to fetch the files from keep-web.

To maximize reuse, dependencies of each CommandLineTool are still copied to separate collections.


Related issues

Related to Arvados - Idea #15580: [CWL] Register workflow and run from git repoNewActions
Actions #1

Updated by Peter Amstutz over 4 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz over 4 years ago

  • Description updated (diff)
  • Status changed from In Progress to New
Actions #3

Updated by Peter Amstutz over 4 years ago

  • Related to Idea #15580: [CWL] Register workflow and run from git repo added
Actions #4

Updated by Peter Amstutz over 4 years ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz over 4 years ago

  • Description updated (diff)
Actions #6

Updated by Tom Morris over 4 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints
  • Story points set to 3.0
Actions #7

Updated by Peter Amstutz over 2 years ago

  • Target version deleted (Arvados Future Sprints)
Actions #8

Updated by Peter Amstutz about 1 year ago

  • Release set to 60
Actions #9

Updated by Peter Amstutz 9 months ago

  • Release deleted (60)
  • Status changed from New to Resolved
Actions

Also available in: Atom PDF