Project

General

Profile

Actions

Idea #6264

closed

[CWL] Prototype Arvados CWL pipeline runner

Added by Brett Smith almost 9 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
1.0

Description

Implement a tool that acts like arv-run-pipeline-instance that accepts CWL workflows as inputs. It creates and monitors Arvados jobs, continuing until the workflow reaches an end state.

Goals:

  • Have an implementation to demonstrate at BOSC.
  • Build a base that can be developed into a workflow runner usable under Crunch v2.

Simplifying assumptions:

  • For now, the runner only supports an active monitoring mode, akin to arv-run-pipeline-instance's default run mode. No need to support switches like --submit or --no-wait.
  • The runner runs on a host where it can start a Docker container to support CWL's scripting engine capabilities—probably an Arvados shell node.
  • The workflow is only expected to succeed if it uses Docker images with bare minimum Arvados support. At the very least, they need to have virtualenv installed, which Crunch can use to install an Arvados SDK. Accepting arbitrary Docker images will probably require more support from Crunch v2 so it's out of scope for now.

Subtasks 3 (0 open3 closed)

Task #6294: Review 6264-cwl-runnerResolvedPeter Amstutz07/02/2015Actions
Task #6365: Write runner using cwltool and run-commandResolvedPeter Amstutz07/01/2015Actions
Task #6464: Review 6264-run-command-task-envResolvedPeter Amstutz07/01/2015Actions

Related issues

Related to Arvados - Idea #4685: [Crunch] CWL prototype workflow runner in ArvadosResolvedPeter Amstutz04/07/2015Actions
Actions

Also available in: Atom PDF