Keep-balance » History » Version 4

« Previous - Version 4/7 (diff) - Next » - Current version
Tom Clegg, 02/14/2014 10:22 AM


Data Manager

The Data Manager enforces policies and generates reports about storage resource usage. The Data Manager interacts with the Keep server and the metadata database. Clients/users do not interact with the Data Manager directly.

See also: Responsibilities:
  • Garbage collector: decide what is eligible for deletion (and some partial order of preference)
  • Replication enforcer: copy and delete blobs in various backing stores to achieve desired replication level
  • Rebalancer: move blobs to redistribute free space and reduce client probes
  • Tell managers how much disk space is being conserved due to CAS
  • Tell managers how much disk space is occupied in a given backing store service
  • Tell managers how disk usage would be affected by modifying storage policy
  • Tell managers how much disk space+time is used (per user, group, node, disk)
  • Tell users when replication/policy specified for a collection is not currently satisfied (and why, for how long, etc)
  • Tell users how much disk space is represented by a given set of collections
  • Tell users how much disk space can be made available by garbage collection
  • Tell users how soon they should expect their cached data to disappear
  • Tell users performance statistics (how fast should I expect my job to read data?)
  • Tell ops where each block was most recently read/written, in case data recovery is needed
  • Tell ops how unbalanced the backing stores are across the cluster
  • Tell ops activity level and performance statistics
  • Tell ops activity level vs. amount of space (how much of the data is being accessed by users?)
  • Tell ops disk performance/error/status trends (and SMART reports) to help identify bad hardware
  • Tell ops history of disk adds, removals, moves
Basic kinds of data in the index:
  • Which blocks are used by which collections (and which collections are valued by which users/groups)
  • Which blocks are stored on which disks
  • Which disks are attached to which nodes
  • Read events
  • Write events
  • Exceptions (checksum mismatch, IO error)

Implementation considerations

Overview
  • REST service
  • API server may cache/proxy some queries
  • API server may redirect some queries
Permissions
  • Support +A tokens like Keep server when accepting collection/blob uuids in request?
  • Require admin api_token for some queries, site-configurable?
Distributed/asynchronous
  • Easy to run multiple keep index services.
  • Most features do not need synchronous operation / real time data.
  • Features that move or delete data should be tied to a single "primary" indexing service (failover event likely requires resetting some state).
  • Substantial disagreement between multiple index services should be easy to flag on admin dashboard.