Common Workflow Language integration with Arvados¶

The main integration is arvados-cwl-runner. This tool is used both as a client to submit a job to run the workflow (arvados-cwl-runner --submit) and for running the workflow directly (arvados-cwl-runner --local). When the user starts a CWL workflow from workbench, it functions similar to --submit mode.

Because submit mode works by creating a "runner" job whose purpose is to manage execution of the workflow, it requires an arvados/jobs Docker image with an up-to-date arvados-cwl-runner.

When running a workflow:

1. The user submits a workflow by running arvados-cwl-runner or through workbench, this creates an "workflow job"
2. The workflow job is dispatched to a compute node (Compute1)
3. The workflow job runs arvados-cwl-runner in a arvados/jobs Docker container with the associated CWL workflow and workflow inputs as job inputs
4. The runner job is responsible for actually executing the workflow:
   1. Create a job for each workflow step
   2. Submit the job to API server (Job A)
   3. Wait for the job to complete
   4. Get the output collection for the job (Output A)
   5. Compute the resulting CWL output object for the workflow step (Result A)
   6. Iterate over the workflow until all steps have been executed and final workflow output is available
5. Return the final workflow result