Arvados

h1. Distributed workflows

h2. Problem description

A user wants to run a meta-analysis on data located on several different clusters.  For either efficiency or legal reasons, the data should be analyzed in place and the results aggregated and returned to a central location.  The user should be able to express the multi-cluster computation as a single CWL workflow, and no manual intervention should be required while the workflow is running.

h2. Simplifying assumptions

User explicitly indicates in the workflow which cluster a certain computation (data+code) happens.

Data transfer only occurs between the primary cluster and the secondary clusters, not between secondary clusters.

h2. Proposed solution

A workflow step can be given a CWL hint "RunOnCluster".  This indicates the tool or subworkflow run by the workflow step should run on a specific Arvados cluster, instead of submitted to the cluster that the workflow runner is currently running on.  The implementation of this would be similar to the "RunInSingleContainer" feature, constructing a container request to run the workflow runner on the remote cluster and wait for results.

In order for the workflow to run successfully on the remote cluster, it needs its data dependencies (docker images, scripts, reference data, etc).  These are several options:

# Don't do any data transfer of dependencies.  Workflows will fail if dependencies are not available.  User must manually transfer collections using arv-copy.

** pros: least work

** cons: terrible user experience.  workflows that involve transferring intermediate results out to remote clusters don't work.

# Distribute dependencies as part of workflow registration (requires proactively distributing dependencies to every cluster that might ever need it).

** pros: less burden on user than option (1)

** cons: doesn't guarantee the dependencies are available where needed, --create/update-workflow option of arvados-cwl-runner has to orchestrate upload of data to every cluster in the federation

# Workflow runner determines which dependencies are missing from the remote cluster and pushes them before scheduling the subworkflow.

** pros: no user intervention required, only copy data to clusters that we think will need it

** cons: copies all dependencies regardless of whether they are actually used, requires that the primary runner have all the dependencies, or is able to facilitate transfer from some other cluster

* Workflow runner on remote cluster determines which dependencies are missing and pulls them from federated peers on demand.

** pros: no user intervention required, only copy data we actually need

** cons: requires that the primary runner have all the dependencies, or is able to facilitate transfer from some other cluster

* Federated access to collections, fetch data blocks on demand from another cluster

** pros: only fetch data blocks that are actually needed, no collection record in the database

** cons: requires federation infrastructure that isn't designed yet, requires some sort of caching proxy to avoid re-fetching the same block (for example, if 100 nodes are all trying to run a docker image from a federated collection).

Options 2, 3 and 4 require roughly equivalent development work.