Arvados

h1. Keep storage pools

Related to (but not the same thing as) [[Keep storage tiers]]. For some use cases, the assumption of a roughly linear relationship between slow/cheap and fast/expensive doesn't necessarily hold.  

Propose that storage pools are independent.  There is no implied relationship between pools.  Pools can be identified with labels or uuids instead of integers.  The keep services table adds a column which lists which pools are available at which services.

When writing blocks, keepstore recognizes a header @X-Keep-Pool@ and accepts or denies the block based on whether it can place the block in the designated pool.  If not supplied, keepstores should have a default pool.  The value of @X-Keep-Pool@ should be reported in the response.

A keepstore mount is associated with a specific pool.

Collections specify the desired pool for the blocks in the collection.  Keep balance should move blocks to the desired pool.  If multiple collections reference the same block in different pools, each pool should have a copy.

Data management policies, for example "move data from hot storage to cold storage if not accessed after 1 month", should be implemented with additional tooling/scripts on top of the keepstore later.