Project

General

Profile

Keep manifest format » History » Version 2

Tom Clegg, 11/17/2014 03:02 PM

1 1 Tom Clegg
h1. Keep manifest format
2
3
h2. Manifest v1
4
5 2 Tom Clegg
A manifest is utf-8 encoded text, consisting of one or more newline-terminated streams.
6 1 Tom Clegg
7
Each stream consists of three or more space-delimited tokens:
8
* The first token is a stream name, consisting of one or more tokens, delimited by @"/"@, the first of which is always @"."@.
9
* The second token is a data blob locator, consisting of one or more tokens, delimited by @"+"@, the first of which is an MD5 hexdigest.
10
** If a subsequent token ("hint") in the locator is numeric, it indicates the size of the data blob, in bytes.
11
** If a hint starts with @"A"@, it is an authorization token (used by the Keep server to confirm that the block is readable by a specific API auth token).
12
* ...possibly followed by more data blob locators...
13
* The first token that is not a block locator, and all subsequent tokens, are file tokens.
14
** A file token has three parts, delimited by @":"@: position, size, filename.
15
** Position and size are given in decimal, and are counted from the beginning of the first data blob.
16
** Filename may contain @"/"@ characters, but must not start or end with @"/"@, and must not contain @"//"@.
17
18
h2. Normalized manifest v1
19
20
A normalized manifest has the following additional restrictions.
21
* Streams are in alphanumeric order.
22
* Each stream name is unique within the manifest.
23
* Files within a stream are in alphanumeric order.
24
* -Concatenation @stream_name/filename@ is unique within the manifest.- (This can be impossible to accomplish without rewriting blobs.)
25
* Filename must not contain @"/"@.
26
27
An API call -exists- will exist to normalize a manifest.
28
29
Request:
30
* @POST /arvados/v1/collections/{hash}/normalize@
31
* request body: @{"collection":{"manifest_text":"...."}}@
32
33
Response:
34
* @{"uuid":"...","manifest_text":"..."}@
35
36
Notes:
37
* POST despite no side effects.
38
* Returns object with uuid even though no object was stored.