Arvados

[THIS IS NO LONGER RELEVANT]

The main challenges are

1. return responses as we get them (so one slow site doesn't slow down the whole response)
2. stateless paging

Items in the response can only be globally ordered by waiting for and merging responses from every site. We can restrict/redefine ordering behavior, such as: objects on the same cluster will be returned sequentially and ordered internally, the client is expected to re-sort.

If we write the response json object incrementally (as remote responses come back) we could use streaming (event driven) parsing in the browser with http://oboejs.com/

Note: writing incrementally probably means using chunked Transfer-Encoding. See also https://github.com/jonnyreeves/chunked-request

On paging.

Given the ordering restriction described above, the client could use a recipe like:

For each sequence of items on the same cluster, look for the last one (as defined by a cut-over in the list from one uuid prefix to another, or the end of the list). Use each "final" item to construct a query that fetches the next page of items.

Note: we might not know that we have the "last" item if the following response is delayed, so we are delayed in seeing the cut-over. Maybe "items" should be a list of lists, so we see the termination of the list instead? Or maybe "items" should be a map of cluster → items.

Maybe a federated response special form:

{
  "kind": "arvados#federatedList",
  "parts": {
    "qr1hi": {
      "items": [...]
    },
    "4xphq": {
      "items": [...]
    }
  }
}

With corresponding federated query?

filters: [
  ["any", "@@", "my query string"], 
  {
    "parts": {
       "qr1hi": [["uuid", ">", "123"]], 
       "4xphq": [["uuid", ">", "456"]]
   }
] 

Strike note-2 in favor of current description.