Federation implementation roadmap » History » Version 8

Version 7 (Peter Amstutz, 08/08/2018 04:04 PM) → Version 8/16 (Tom Clegg, 08/21/2018 06:14 PM)

h1. Federation implementation roadmap

Use a remote-hosted workflow #13493

* When retrieving a workflow record for arvados-cwl-runner, API server notices workflow UUID is remote, and fetches it with salted token (instead of looking in local DB)

Mount remote-hosted collection (read-only) - [[Federated collections]]

* (controller) fetch remote-hosted container by UUID (similar to workflow case) (#13993)
* (controller) rewrite locators in manifest to indicate remote cluster (#13993)
* (keepstore) if locator indicates remote cluster and data does not exist locally (or local signature is not supplied), fetch from remote using salted token (#13994)

Mount remote-hosted collection (read-only) in a container

* When creating a container to satisfy a CR, Rails API assumes any searches for collection PDH appearing in the CR mounts is OK.
* When running the container, arv-mount (and any other client) will go through the controller to get the collection,
cr.mounts across federated nodes, and the controller will search the federation for any PDH not found (or not readable by the acting user) fetches it with salted token (instead of looking in the local DB. DB)

Run some portions of a CWL workflow on a remote cluster

* When submitting container requests, a-c-r follows workflow/commandline/input UUID hints to choose a cluster
* Either a-c-r handles the multi-cluster stuff itself, or it submits CRs with "submit to cluster X" flag, and the API server acts as a proxy