Keep » History » Version 1

Tom Clegg, 04/10/2013 04:23 PM

1 1 Tom Clegg
h1. Keep
2 1 Tom Clegg
3 1 Tom Clegg
The Keep storage system consists of data block read/write services, SDKs, and management agents.
4 1 Tom Clegg
5 1 Tom Clegg
The responsibilities of the Keep service are:
6 1 Tom Clegg
7 1 Tom Clegg
* Write data blocks
8 1 Tom Clegg
* When writing: ensure data integrity by comparing client-supplied MD5 digest to client-supplied data
9 1 Tom Clegg
* Read data blocks (subject to permission, which is determined by the system/metadata DB)
10 1 Tom Clegg
* Send read/write/error event logs to management agents
11 1 Tom Clegg
12 1 Tom Clegg
The responsibilities of the SDK are:
13 1 Tom Clegg
14 1 Tom Clegg
* When writing: split data into ≤64 MiB chunks
15 1 Tom Clegg
* When writing: encode directory trees as manifests
16 1 Tom Clegg
* When writing: write data to the desired number of nodes to achieve storage redundancy
17 1 Tom Clegg
* After writing: register a collection with Arvados
18 1 Tom Clegg
* When reading: parse manifests
19 1 Tom Clegg
* When reading: verify data integrity by comparing locator to MD5 digest of retrieved data