Idea #3414
Updated by Tom Clegg over 9 years ago
The keep server should accept a json file at PUT /pull This file will contain an ordered list of of block locators for blocks to pull from other keep servers. For each block, it will contain an ordered list of keep server addresses (host & port) to pull the blocks from. When the keep server has resources available, it should request these blocks in order, trying the servers in the order given, and store the block returned in its own storage. Each pull list received overwrites all previous pull lists. This is written out at the bottom of: https://arvados.org/projects/orvos-private/wiki/Keep_Design_Doc#Keep-Server Prioritization: # Accept a list of blocks-to-pull. # Log the new list on stderr. # Replace the current list in memory (if any) with the new list. # Process the list by pulling data from remote servers. Implementation notes: * Currently keepstore does not check whether the client's API token itself is valid. Therefore, when a keepstore retrieves blocks from other keepstores, it can use a made-up API token (like a string of fifty "x") in the Authorization header of its request (and, of course, use the same token to generate a permission hint). (see #3414#note-8 note !#8 below for more detail) * Suggest: one goroutine is dedicated to owning the current pull list. Other goroutines use its channel to perform operations like "replace list with this one", "get list", "remove data block X from list". Possibly useful example: https://gobyexample.com/stateful-goroutines