Keep-balance » History » Version 6
Tom Clegg, 03/16/2017 08:56 PM
| 1 | 6 | Tom Clegg | h1. Keep-balance |
|---|---|---|---|
| 2 | 1 | Tom Clegg | |
| 3 | 6 | Tom Clegg | Keep-balance enforces policies and generates reports about storage resource usage. It interacts with the [[Keep server]] and the metadata database (through the API server). Clients/users do not interact with keep-balance directly. |
| 4 | 1 | Tom Clegg | |
| 5 | 4 | Tom Clegg | See also: |
| 6 | * [[Keep server]] |
||
| 7 | * [[Keep manifest format]] |
||
| 8 | 6 | Tom Clegg | * source:services/keep-balance |
| 9 | 1 | Tom Clegg | |
| 10 | 4 | Tom Clegg | Responsibilities: |
| 11 | 5 | Tom Clegg | * Garbage collector: decide what is eligible for deletion (and some partial order of preference) |
| 12 | * Replication enforcer: copy and delete blocks in various backing stores to achieve desired replication level |
||
| 13 | * Rebalancer: move blocks to redistribute free space and reduce client probes |
||
| 14 | 4 | Tom Clegg | |
| 15 | 5 | Tom Clegg | Example reports/queries: |
| 16 | * for managers: how much disk space is being conserved due to CAS |
||
| 17 | * for managers: how much disk space is occupied in a given backing store service |
||
| 18 | * for managers: how disk usage would be affected by modifying storage policy |
||
| 19 | * for managers: how much disk space+time is used (per user, group, node, disk) |
||
| 20 | * for users: when replication/policy specified for a collection is not currently satisfied (and why, for how long, etc) |
||
| 21 | * for users: how much disk space is represented by a given set of collections |
||
| 22 | * for users: how much disk space can be made available by garbage collection |
||
| 23 | * for users: how soon they should expect their cached data to disappear |
||
| 24 | * for users: performance statistics (how fast should I expect my job to read data?) |
||
| 25 | * for ops: where each block was most recently read/written, in case data recovery is needed |
||
| 26 | * for ops: how unbalanced the backing stores are across the cluster |
||
| 27 | * for ops: activity level and performance statistics |
||
| 28 | * for ops: activity level vs. amount of space (how much of the data is being accessed by users?) |
||
| 29 | * for ops: disk performance/error/status trends (and SMART reports) to help identify bad hardware |
||
| 30 | * for ops: history of disk adds, removals, moves |
||
| 31 | |||
| 32 | 4 | Tom Clegg | Basic kinds of data in the index: |
| 33 | * Which blocks are used by which collections (and which collections are valued by which users/groups) |
||
| 34 | 5 | Tom Clegg | * Which blocks are stored in which services (local Keep, remote Keep, other storage service) |
| 35 | 4 | Tom Clegg | * Which blocks are stored on which disks |
| 36 | * Which disks are attached to which nodes |
||
| 37 | 5 | Tom Clegg | * Aggregate read/write activity per block and per disk (where applicable, e.g., block stored in local Keep) |
| 38 | 4 | Tom Clegg | * Exceptions (checksum mismatch, IO error) |
| 39 | |||
| 40 | h2. Implementation considerations |
||
| 41 | |||
| 42 | Overview |
||
| 43 | 5 | Tom Clegg | * REST service for queries |
| 44 | ** All requests require authentication. Token validity verified against Metadata server, and cached locally. |
||
| 45 | * Subscribes to system event log |
||
| 46 | * Connects to metadata server (has a system_user token), at least periodically, to ensure eventual consistency with metadata DB's idea of what data is important |
||
| 47 | * Persistent database |
||
| 48 | * In-memory database |
||
| 49 | 4 | Tom Clegg | |
| 50 | Distributed/asynchronous |
||
| 51 | * Easy to run multiple keep index services. |
||
| 52 | * Most features do not need synchronous operation / real time data. |
||
| 53 | * Features that move or delete data should be tied to a single "primary" indexing service (failover event likely requires resetting some state). |
||
| 54 | * Substantial disagreement between multiple index services should be easy to flag on admin dashboard. |