Keep-balance » History » Version 6

Tom Clegg, 03/16/2017 08:56 PM

1 6 Tom Clegg
h1. Keep-balance
2 1 Tom Clegg
3 6 Tom Clegg
Keep-balance enforces policies and generates reports about storage resource usage. It interacts with the [[Keep server]] and the metadata database (through the API server). Clients/users do not interact with keep-balance directly.
4 1 Tom Clegg
5 4 Tom Clegg
See also:
6 4 Tom Clegg
* [[Keep server]]
7 4 Tom Clegg
* [[Keep manifest format]]
8 6 Tom Clegg
* source:services/keep-balance
9 1 Tom Clegg
10 4 Tom Clegg
Responsibilities:
11 5 Tom Clegg
* Garbage collector: decide what is eligible for deletion (and some partial order of preference)
12 5 Tom Clegg
* Replication enforcer: copy and delete blocks in various backing stores to achieve desired replication level
13 5 Tom Clegg
* Rebalancer: move blocks to redistribute free space and reduce client probes
14 4 Tom Clegg
15 5 Tom Clegg
Example reports/queries:
16 5 Tom Clegg
* for managers: how much disk space is being conserved due to CAS
17 5 Tom Clegg
* for managers: how much disk space is occupied in a given backing store service
18 5 Tom Clegg
* for managers: how disk usage would be affected by modifying storage policy
19 5 Tom Clegg
* for managers: how much disk space+time is used (per user, group, node, disk)
20 5 Tom Clegg
* for users: when replication/policy specified for a collection is not currently satisfied (and why, for how long, etc) 
21 5 Tom Clegg
* for users: how much disk space is represented by a given set of collections
22 5 Tom Clegg
* for users: how much disk space can be made available by garbage collection
23 5 Tom Clegg
* for users: how soon they should expect their cached data to disappear
24 5 Tom Clegg
* for users: performance statistics (how fast should I expect my job to read data?)
25 5 Tom Clegg
* for ops: where each block was most recently read/written, in case data recovery is needed
26 5 Tom Clegg
* for ops: how unbalanced the backing stores are across the cluster
27 5 Tom Clegg
* for ops: activity level and performance statistics
28 5 Tom Clegg
* for ops: activity level vs. amount of space (how much of the data is being accessed by users?)
29 5 Tom Clegg
* for ops: disk performance/error/status trends (and SMART reports) to help identify bad hardware
30 5 Tom Clegg
* for ops: history of disk adds, removals, moves
31 5 Tom Clegg
32 4 Tom Clegg
Basic kinds of data in the index:
33 4 Tom Clegg
* Which blocks are used by which collections (and which collections are valued by which users/groups)
34 5 Tom Clegg
* Which blocks are stored in which services (local Keep, remote Keep, other storage service)
35 4 Tom Clegg
* Which blocks are stored on which disks
36 4 Tom Clegg
* Which disks are attached to which nodes
37 5 Tom Clegg
* Aggregate read/write activity per block and per disk (where applicable, e.g., block stored in local Keep)
38 4 Tom Clegg
* Exceptions (checksum mismatch, IO error)
39 4 Tom Clegg
40 4 Tom Clegg
h2. Implementation considerations
41 4 Tom Clegg
42 4 Tom Clegg
Overview
43 5 Tom Clegg
* REST service for queries
44 5 Tom Clegg
** All requests require authentication. Token validity verified against Metadata server, and cached locally.
45 5 Tom Clegg
* Subscribes to system event log
46 5 Tom Clegg
* Connects to metadata server (has a system_user token), at least periodically, to ensure eventual consistency with metadata DB's idea of what data is important
47 5 Tom Clegg
* Persistent database
48 5 Tom Clegg
* In-memory database
49 4 Tom Clegg
50 4 Tom Clegg
Distributed/asynchronous
51 4 Tom Clegg
* Easy to run multiple keep index services.
52 4 Tom Clegg
* Most features do not need synchronous operation / real time data.
53 4 Tom Clegg
* Features that move or delete data should be tied to a single "primary" indexing service (failover event likely requires resetting some state).
54 4 Tom Clegg
* Substantial disagreement between multiple index services should be easy to flag on admin dashboard.