Arvados

h1. Distributed workflows

h2. Problem description

A user wants to run a meta-analysis on data located on several different clusters.  For either efficiency or legal reasons, the data should be analyzed in place and the results aggregated and returned to a central location.  The user should be able to express the multi-cluster computation as a single CWL workflow, and no manual intervention should be required while the workflow is running.

h2. Simplifying assumptions

User explicitly indicates in the workflow which cluster a certain computation (data+code) happens.

Data transfer only occurs between the primary cluster and the secondary clusters, not between secondary clusters.

h2. Proposed solution

A workflow step can be given a CWL hint "RunOnCluster".  This indicates the tool or subworkflow run by the workflow step should run on a specific Arvados cluster, instead of submitted to the cluster that the workflow runner is currently running on.  The implementation of this would be similar to the "RunInSingleContainer" feature, constructing a container request to run the workflow runner on the remote cluster and wait for results.

In order for the workflow to run successfully on the remote cluster, it needs its data dependencies (docker images, scripts, reference data, etc).  These are several options:

* Don't do any data transfer of dependencies.  Workflows will fail if dependencies are not available.  User must manually transfer collections using arv-copy.

* Distribute dependencies as part of workflow registration (requires proactively distributing dependencies to every cluster that might ever need it).

* Workflow runner determines which dependencies are missing from the remote cluster and pushes them before scheduling the subworkflow (requires that the primary runner have all the data, or is able to facilitate transfer from some other cluster).

* Workflow runner on remote cluster determines which dependencies are missing and pulls them from federated peers on demand.

... wip