Project

General

Profile

Keep storage classes » History » Version 3

Peter Amstutz, 06/13/2017 08:36 PM

1 1 Tom Clegg
h1. Keep storage pools
2
3 2 Peter Amstutz
h2. Use cases
4
5
* User has option to store some data in cheaper storage, but only certain data qualifies.  Can be indicated on a per-collection basis.
6
* User wants data moved from "hot" to "cool" storage a certain amount of time after it has been generated.
7
8 3 Peter Amstutz
h2. Design
9 2 Peter Amstutz
10 3 Peter Amstutz
Related to (but not the same thing as) [[Keep storage tiers]]. For some use cases, the assumption of a roughly linear relationship between slow/cheap and fast/expensive doesn't necessarily hold.
11 1 Tom Clegg
12 3 Peter Amstutz
Each service has access to one or more storage pools.  Storage pools are independent.  There is no implied relationship between pools.  Data assigned to a pool may still be sharded among multiple servers.  Pools can be identified with labels or uuids instead of integers.  The keep services table adds a column which lists which pools are available at which services.
13 1 Tom Clegg
14
When writing blocks, keepstore recognizes a header @X-Keep-Pool@ and accepts or denies the block based on whether it can place the block in the designated pool.  If not supplied, keepstores should have a default pool.  The value of @X-Keep-Pool@ should be reported in the response.
15
16
A keepstore mount is associated with a specific pool.
17
18
Collections specify the desired pool for the blocks in the collection.  Keep balance should move blocks to the desired pool.  If multiple collections reference the same block in different pools, each pool should have a copy.
19
20
Data management policies, for example "move data from hot storage to cold storage if not accessed after 1 month", should be implemented with additional tooling/scripts on top of the keepstore later.