Common Workflow Language integration with Arvados¶

The main integration is arvados-cwl-runner. This tool is used both as a client to submit a job to run the workflow (arvados-cwl-runner --submit) and for running the workflow directly (arvados-cwl-runner --local). When the user starts a CWL workflow from workbench, it functions similar to --submit mode.

Because submit mode works by creating a "runner" job whose purpose is to manage execution of the workflow, it requires an arvados/jobs Docker image with an up-to-date arvados-cwl-runner.

When running a workflow:

1. The user submits a workflow by running arvados-cwl-runner or through workbench, this creates an "workflow orchestrator"
2. The workflow orchestrator is dispatched to a compute node (Compute1)
3. The workflow orchestrator runs arvados-cwl-runner in a arvados/jobs Docker container with the associated CWL workflow and workflow inputs as job inputs
4. The orchestrator job is responsible for actually executing the workflow:
   1. Create a container request (job) for each workflow step
   2. Submit the job to API server (Container A)
   3. Wait for the job to complete
   4. Get the output collection for the job (Output A)
   5. Compute the resulting CWL output object for the workflow step (Result A)
   6. Iterate over the workflow until all steps have been executed and final workflow output is available
5. Return the final workflow result