Project

General

Profile

Actions

Idea #3414

closed

[Keep] Keep Server accepts list of blocks to pull from other servers

Added by Misha Zatsman over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tim Pierce
Category:
Keep
Target version:
Story points:
2.0

Description

The keep server should accept a json file at PUT /pull

This file will contain an ordered list of of block locators for blocks to pull from other keep servers.

For each block, it will contain an ordered list of keep server addresses (host & port) to pull the blocks from.

When the keep server has resources available, it should request these blocks in order, trying the servers in the order given, and store the block returned in its own storage.

Each pull list received overwrites all previous pull lists.

This is written out at the bottom of:
https://arvados.org/projects/orvos-private/wiki/Keep_Design_Doc#Keep-Server

Prioritization:
  1. Accept a list of blocks-to-pull.
  2. Log the new list on stderr.
  3. Replace the current list in memory (if any) with the new list.
  4. Process the list by pulling data from remote servers.
Implementation notes:
  • Currently keepstore does not check whether the client's API token itself is valid. Therefore, when a keepstore retrieves blocks from other keepstores, it can use a made-up API token (like a string of fifty "x") in the Authorization header of its request (and, of course, use the same token to generate a permission hint). (see #3414#note-8 note #8 below for more detail)
  • Suggest: one goroutine is dedicated to owning the current pull list. Other goroutines use its channel to perform operations like "replace list with this one", "get list", "remove data block X from list". Possibly useful example: https://gobyexample.com/stateful-goroutines

Subtasks 2 (0 open2 closed)

Task #3697: Review 3414-keep-pull-handlerResolvedTom Clegg08/21/2014Actions
Task #3650: Implement PullHandlerResolvedTim Pierce08/21/2014Actions

Related issues

Blocks Arvados - Idea #3408: [Keep] Implement Production Data ManagerResolvedPeter Amstutz07/29/2014Actions
Actions

Also available in: Atom PDF