Story #15535

[CWL] Run from original CWL, not packed

Added by Peter Amstutz 3 months ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
3.0

Description

arvados-cwl-runner, on submitting a workflow, uses "pack" to create a single-stream document.

This is because the "workflow" record used to display workflows on workbench only stores a single raw text field, into which the multi-document CWL file has to be stuffed. The rationale for having a workflow record be a text field and not a PDH or git hash to avoid requiring workbench be able to fetch a collection / git repo to display a workflow. Although this isn't a limitation when submitting from the command line, this also uses the "pack" function to minimize having multiple code paths.

Unfortunately the packed version often bares little resemblance to user's original document. It would be better to execute the original document.

Proposal:

At CLI: Upload original workflow files & dependencies to a collection, preserving original filesystem structure. Submit a container request that mounts the collection and runs the workflow.

At workbench: to register workflow record, create a wrapper workflow that has the same input/output interface as the workflow, with a single step with a run line like:

run: keep:pdh/workflow

To submit the workflow, workbench introspects the step and sets up the correct collection mount.

To display cwl-svg of a workflow, workbench2 needs to be able to fetch the files from keep-web.

To maximize reuse, dependencies of each CommandLineTool are still copied to separate collections.


Related issues

Related to Arvados - Story #15580: [CWL] Register workflow and run from git repoNew

History

#1 Updated by Peter Amstutz 3 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 3 months ago

  • Status changed from In Progress to New
  • Description updated (diff)

#3 Updated by Peter Amstutz 3 months ago

  • Related to Story #15580: [CWL] Register workflow and run from git repo added

#4 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#6 Updated by Tom Morris 3 months ago

  • Story points set to 3.0
  • Target version changed from To Be Groomed to Arvados Future Sprints

Also available in: Atom PDF