Keep » History » Version 1

Version 1/26 - Next » - Current version
Tom Clegg, 04/10/2013 04:23 PM


Keep

The Keep storage system consists of data block read/write services, SDKs, and management agents.

The responsibilities of the Keep service are:

  • Write data blocks
  • When writing: ensure data integrity by comparing client-supplied MD5 digest to client-supplied data
  • Read data blocks (subject to permission, which is determined by the system/metadata DB)
  • Send read/write/error event logs to management agents

The responsibilities of the SDK are:

  • When writing: split data into ≤64 MiB chunks
  • When writing: encode directory trees as manifests
  • When writing: write data to the desired number of nodes to achieve storage redundancy
  • After writing: register a collection with Arvados
  • When reading: parse manifests
  • When reading: verify data integrity by comparing locator to MD5 digest of retrieved data