Project

General

Profile

Federation implementation roadmap » History » Revision 11

Revision 10 (Tom Clegg, 08/21/2018 06:33 PM) → Revision 11/16 (Tom Clegg, 08/21/2018 07:15 PM)

h1. Federation implementation roadmap 

 Use a remote-hosted workflow #13493 

 * When retrieving a workflow record for arvados-cwl-runner, API server notices workflow UUID is remote, and fetches it with salted token (instead of looking in local DB) 

 Mount remote-hosted collection (read-only) - [[Federated collections]] 

 * (controller) fetch remote-hosted container by UUID (similar to workflow case) (#13993) 
 * (controller) rewrite locators in manifest to indicate remote cluster    (#13993) 
 * (keepstore) if locator indicates remote cluster and data does not exist locally (or local signature is not supplied), fetch from remote using salted token (#13994) 

 Mount remote-hosted collection (read-only) in a container 

 * When creating a container to satisfy a CR, Rails API assumes any PDH appearing in the CR mounts is OK. 
 * When running the container, arv-mount (and any other client) will go through the controller to get the collection. 
 * The controller will search the federation for any PDH not found (or not readable by the acting user) in the local DB. (#14087) 

 Run some portions of a CWL workflow on a remote cluster 

 * When submitting container requests, a-c-r follows workflow/commandline/input UUID hints to choose a cluster. 
 * a-c-r submits CRs to its default cluster with "submit to cluster X" flag, and the controller proxies those requests to remote clusters. 
 * When a-c-r requests a batch of containers, the controller proxies requests to remotes as needed to get a full response (#13619) 
 * This might require changes/additions to the list API