Keep manifest format » History » Version 4

Tom Clegg, 06/11/2015 06:09 PM

1 3 Tom Clegg
{{toc}}
2 3 Tom Clegg
3 1 Tom Clegg
h1. Keep manifest format
4 1 Tom Clegg
5 1 Tom Clegg
h2. Manifest v1
6 1 Tom Clegg
7 2 Tom Clegg
A manifest is utf-8 encoded text, consisting of one or more newline-terminated streams.
8 1 Tom Clegg
9 1 Tom Clegg
Each stream consists of three or more space-delimited tokens:
10 4 Tom Clegg
* The first token is a stream name, consisting of one or more tokens, delimited by @"/"@, the first of which is always @"."@ and none of which are empty. The stream name never begins or ends with @"/"@.
11 1 Tom Clegg
* The second token is a data blob locator, consisting of one or more tokens, delimited by @"+"@, the first of which is an MD5 hexdigest.
12 1 Tom Clegg
** If a subsequent token ("hint") in the locator is numeric, it indicates the size of the data blob, in bytes.
13 1 Tom Clegg
** If a hint starts with @"A"@, it is an authorization token (used by the Keep server to confirm that the block is readable by a specific API auth token).
14 1 Tom Clegg
* ...possibly followed by more data blob locators...
15 1 Tom Clegg
* The first token that is not a block locator, and all subsequent tokens, are file tokens.
16 1 Tom Clegg
** A file token has three parts, delimited by @":"@: position, size, filename.
17 1 Tom Clegg
** Position and size are given in decimal, and are counted from the beginning of the first data blob.
18 1 Tom Clegg
** Filename may contain @"/"@ characters, but must not start or end with @"/"@, and must not contain @"//"@.
19 1 Tom Clegg
20 1 Tom Clegg
h2. Normalized manifest v1
21 1 Tom Clegg
22 1 Tom Clegg
A normalized manifest has the following additional restrictions.
23 1 Tom Clegg
* Streams are in alphanumeric order.
24 1 Tom Clegg
* Each stream name is unique within the manifest.
25 1 Tom Clegg
* Files within a stream are in alphanumeric order.
26 1 Tom Clegg
* -Concatenation @stream_name/filename@ is unique within the manifest.- (This can be impossible to accomplish without rewriting blobs.)
27 1 Tom Clegg
* Filename must not contain @"/"@.
28 1 Tom Clegg
29 1 Tom Clegg
An API call -exists- will exist to normalize a manifest.
30 1 Tom Clegg
31 1 Tom Clegg
Request:
32 1 Tom Clegg
* @POST /arvados/v1/collections/{hash}/normalize@
33 1 Tom Clegg
* request body: @{"collection":{"manifest_text":"...."}}@
34 1 Tom Clegg
35 1 Tom Clegg
Response:
36 1 Tom Clegg
* @{"uuid":"...","manifest_text":"..."}@
37 1 Tom Clegg
38 1 Tom Clegg
Notes:
39 1 Tom Clegg
* POST despite no side effects.
40 1 Tom Clegg
* Returns object with uuid even though no object was stored.
41 3 Tom Clegg
42 3 Tom Clegg
h2. Manifest v2
43 3 Tom Clegg
44 3 Tom Clegg
(Early design stages)
45 3 Tom Clegg
46 3 Tom Clegg
Should probably include:
47 3 Tom Clegg
* Structured format (JSON?)
48 3 Tom Clegg
* More than one level of indirection (e.g., manifest references block X, which references data blocks A,B,C)
49 3 Tom Clegg
* Specify hash algorithm with block hashes