Federated collections » History » Version 2
Peter Amstutz, 08/01/2018 06:58 PM
1 | 1 | Peter Amstutz | h1. Federated collections |
---|---|---|---|
2 | 2 | Peter Amstutz | |
3 | * Fetch collection record by uuid |
||
4 | ** use federated record retrieval strategy, already developed. |
||
5 | * Fetch collection record by PDH |
||
6 | ** No location hint. Must send out request to all federated clusters. |
||
7 | ** Read-only, only need to support GET operation |
||
8 | * Can cache result by PDH. |
||
9 | |||
10 | Record will have a manifest with signed blocks. However these blocks will be signed for the origin cluster. |
||
11 | |||
12 | Client needs to be able to fetch blocks from remote cluster. |
||
13 | |||
14 | arvados-controller could add block hints, using existing feature in the Python and Go SDK: |
||
15 | |||
16 | * Blocks in a manifest can include a hint in the form "+K@zzzzz". Python SDK will attempt to fetch the block from "https://keep.zzzzz.arvadosapi.com/" |
||
17 | ** Must conform to a particular naming DNS scheme. |
||
18 | ** Could be generalized by looking up in "remote_hosts" and using the "keep_services.accessible" API. |
||
19 | ** Every block will be requested from remote every time, because client is contacting remote server directly, limited opportunity for edge caching. |
||
20 | |||
21 | * Hint can also be a uuid of a "local gateway service". This is instructs client to use a specific service from the keep_services table (indicated as "service_type" of "gateway:") |
||
22 | ** Direct requests through a specific service |
||
23 | ** Does not encode which remote cluster to pull a block from. |
||
24 | ** Gateway service could search for blocks by sending request to every federated cluster |
||
25 | ** Gateway service can cache blocks so they don't need to be re-fetched from remote. |