Federation implementation roadmap » History » Version 15

Version 14 (Peter Amstutz, 09/12/2018 02:36 PM) → Version 15/16 (Peter Amstutz, 09/12/2018 02:49 PM)

h1. Federation implementation roadmap

Use a remote-hosted workflow #13493

* When retrieving a workflow record for arvados-cwl-runner, API server notices workflow UUID is remote, and fetches it with salted token (instead of looking in local DB)

Mount remote-hosted collection (read-only) - [[Federated collections]]

* (controller) fetch remote-hosted container by UUID (similar to workflow case) (#13993)
* (controller) rewrite locators in manifest to indicate remote cluster (#13993)
* (keepstore) if locator indicates remote cluster and data does not exist locally (or local signature is not supplied), fetch from remote using salted token (#13994)

Mount remote-hosted collection (read-only) in a container

* When creating a container to satisfy a CR, Rails API assumes any PDH appearing in the CR mounts is OK.
* When running the container, arv-mount (and any other client) will go through the controller to get the collection.
* The controller will search the federation for any PDH not found (or not readable by the acting user) in the local DB. (#14087)
* API server issues v2 format auth tokens (#14196)

Run some portions of a CWL workflow on a remote cluster

* When submitting container requests, a-c-r follows workflow/commandline/input UUID hints to choose a cluster.
* a-c-r submits CRs to its home cluster with owner_uuid, if owner_uuid is a remote cluster, the controller proxies those requests to remote the cluster (#14197)
* When creating a container request on a remote cluster, submitter can specify a container token with appropriate privileges
* When a-c-r requests a batch of containers, the controller proxies requests to remotes as needed to get a full response (#13619)
* To prepare final output collection, must be able to copy files from a remote hosted collection to a local one.
** Requires converting blocks with +R hints to +A hints (#14199)
** Involves copying blocks from remote cluster to local storage to get local signed locator to create collection. (if necessary, could be done on client / cwl-runner side.)