Project

General

Profile

Keep manifest format » History » Version 7

Tom Clegg, 06/01/2016 02:19 PM

1 3 Tom Clegg
{{toc}}
2
3 1 Tom Clegg
h1. Keep manifest format
4
5
h2. Manifest v1
6
7 6 Tom Clegg
A manifest is utf-8 encoded text, consisting of zero or more newline-terminated streams.
8 1 Tom Clegg
9
Each stream consists of three or more space-delimited tokens:
10 5 Tom Clegg
* The first token is a stream name, consisting of one or more path components, delimited by @"/"@.
11
** The first path component is always @"."@.
12
** No path component is empty.
13
** No path component is "." or "..".
14
** The stream name never begins or ends with @"/"@.
15 7 Tom Clegg
* The second token is a data blob locator (see [[Keep locator format]]).
16 1 Tom Clegg
* ...possibly followed by more data blob locators...
17
* The first token that is not a block locator, and all subsequent tokens, are file tokens.
18
** A file token has three parts, delimited by @":"@: position, size, filename.
19
** Position and size are given in decimal, and are counted from the beginning of the first data blob.
20
** Filename may contain @"/"@ characters, but must not start or end with @"/"@, and must not contain @"//"@.
21 5 Tom Clegg
** Filename components (delimited by @"/"@) must not be @"."@ or @".."@.
22
23 1 Tom Clegg
A manifest contains no TAB characters, nor other ASCII whitespace characters other than the spaces or newline delimiters specified above.
24 6 Tom Clegg
25
A manifest always ends with a newline -- except the empty (zero-length) string, which is a valid manifest.
26 1 Tom Clegg
27
h2. Normalized manifest v1
28
29
A normalized manifest has the following additional restrictions.
30
* Streams are in alphanumeric order.
31
* Each stream name is unique within the manifest.
32
* Files within a stream are in alphanumeric order.
33
* -Concatenation @stream_name/filename@ is unique within the manifest.- (This can be impossible to accomplish without rewriting blobs.)
34
* Filename must not contain @"/"@.
35
36
An API call -exists- will exist to normalize a manifest.
37
38
Request:
39
* @POST /arvados/v1/collections/{hash}/normalize@
40
* request body: @{"collection":{"manifest_text":"...."}}@
41
42
Response:
43
* @{"uuid":"...","manifest_text":"..."}@
44
45
Notes:
46
* POST despite no side effects.
47
* Returns object with uuid even though no object was stored.
48 3 Tom Clegg
49
h2. Manifest v2
50
51
(Early design stages)
52
53
Should probably include:
54
* Structured format (JSON?)
55
* More than one level of indirection (e.g., manifest references block X, which references data blocks A,B,C)
56
* Specify hash algorithm with block hashes