Arvados

h1. Keep service hints

h3. Objective

Clients can use the Keep API to retrieve data that is stored on remote servers. As with local data, permission tokens and data hashes are provided by the API server in a manifest.

h3. Background

When a client reads a data block referenced by a manifest, it requests a list of "available keep services" from the API server and (if there is more than one "disk" service on the list) uses the rendezvous hashing algorithm to select one.

h3. Alternatives

Client libraries could communicate directly with non-Keep services.

* It would be impossible to use Arvados permission controls.

* An N×M array of code would have to be maintained in order to support N backing services from M SDK languages.

* The API server would have to maintain the mapping of hashes to remote data objects (and permissions for this map).

Each keepstore server could know how to communicate with each non-Keep service in use.

* Simpler client code.

* Artificial link between keep disk services and gateway services (they couldn't be independently scaled or shut down for maintenance).

* External clients couldn't be given direct access to the third-party gateway services without also giving them direct access to the disk services.

* Either the keepstore servers would have to keep their hash-to-remote-object mappings synchronized -- or the map of hash to remote service would be distributed across various servers. Either way introduces an unsuitable level of complexity: unlike in a native keepstore system, the underlying data is expected to change over time.

* When encountering an error (notably 404), client code would make many redundant attempts to read from various gateway services, based on the mistaken assumption that the various services have different sets of available data blocks.

h3. High level design

Tools (TBD) can create manifests with @+Kuuid@ hints, referencing data in remote storage services by indicating the UUID of a storage gateway capable of accessing it. _In future work, Arvados can manage these hints actively: for example, data manager could tag blocks with S3 bucket names, and API server could load-balance S3 gateways by selecting one of several available gateway UUIDs for a given block._

Each client library knows that when it sees @+Kuuid@ it should connect to the keep service with the given UUID (instead of using the usual rendezvous hashing algorithm to select a service).

h2. Specifics

A block locator provided by the API server in a manifest might have a hint of the form @+Kuuid@ where @uuid@ is the UUID of a keep service. In order to retrieve the block data, the client should look up the keep service with the given UUID, and perform an HTTP @GET@ request at the appropriate host and port.

* Given @acbd18db4cc2f85cedef654fccc4a4d8+3+K1h9kt-bi6l4-20fty0xbp8l9wwe@,

** Retrieve @https://1h9kt.arvadosapi.com/arvados/v1/keep_services/1h9kt-bi6l4-20fty0xbp8l9wwe@ to determine scheme, host, port

** Retrieve data from @{scheme}://{host}{port}/acbd18db4cc2f85cedef654fccc4a4d8+3+K1h9kt-bi6l4-20fty0xbp8l9wwe@

As before, if a hint of the form @+K{prefix}@ is given (where @{prefix}@ is a string of five characters in @[0-9a-z]@), the client should perform a @GET@ request at @https://keep.{prefix}.arvadosapi.com/locator@.

* Given @acbd18db4cc2f85cedef654fccc4a4d8+3+K1h9kt@,

** Retrieve data from @https://keep.1h9kt.arvadosapi.com/acbd18db4cc2f85cedef654fccc4a4d8+3+K1h9kt@

h2. Future work

Data manager can update manifests to reflect additional locations where data blocks can be retrieved: for example, @+Kuuid1+Kuuid2@ to signify that multiple remote gateways can retrieve the data, or @+K+Kuuid1@ to signify that the data is available locally _and_ via a remote gateway.