Collection update API » History » Version 3

Peter Amstutz, 05/19/2015 01:26 PM

1 1 Peter Amstutz
h1. Collection update API
2 1 Peter Amstutz
3 1 Peter Amstutz
Problem: adding, updating, renaming, or removing a file in a collection requires sending the entire manifest text of the collection.  This approach creates two problems:
4 1 Peter Amstutz
5 1 Peter Amstutz
# It is inefficient; resources are wasted transferring and processing manifest data that hasn't changed 
6 2 Peter Amstutz
# It creates a race condition; for example processes A and B read the collection record at time T1, merge it with their local updates, process A sends its update at time T2, process B sends its update at time T3; this causes process A's updates to be dropped.
7 1 Peter Amstutz
8 3 Peter Amstutz
h2. Proposed solution
9 1 Peter Amstutz
10 3 Peter Amstutz
Introduce new 'update' or 'patch' endpoint for collections supporting a patching protocol allowing incremental collection update.
11 3 Peter Amstutz
12 2 Peter Amstutz
The protocol consists of two fields: *remove* and *add*.  Each field contains a fragment of manifest text.
13 1 Peter Amstutz
14 2 Peter Amstutz
The *remove* field lists files that should be removed from the collection.  Remove is processed before add.
15 2 Peter Amstutz
For each file, check that the file in the collection has the same content (== same blocks/segments) as the file to be removed.  If the file contents match, the file is removed.  If the file in tho collection file does not match the file to be removed, the file in the collection is renamed to a new unique name that indicates it is in conflict.  If the file is not present in the collection, do nothing.
16 1 Peter Amstutz
17 2 Peter Amstutz
The *add* field lists files that should be added to the manifest.  If a file does not exist in the collection, it is added.  If the file  already exists in the collection, the existing file in the collection is renamed to a new unique name that indicates it is in conflict; the new file is put in its place.
18 1 Peter Amstutz
19 2 Peter Amstutz
After the patch call, respond with the portable data hash.  The caller can compare the PDH to the expected result to determine if there were conflicts that require it to update its local collection contents.
20 2 Peter Amstutz
21 1 Peter Amstutz
Any other files in the collection that are not mentioned in the *remove* or *add* operations are unaffected.
22 3 Peter Amstutz
23 3 Peter Amstutz
h2. Use
24 2 Peter Amstutz
25 2 Peter Amstutz
With this protocol we can implement add, delete, rename, modify, and combine collections:
26 2 Peter Amstutz
27 2 Peter Amstutz
* add: provide one or more files to be added in the *add* field
28 2 Peter Amstutz
* remove: provide one or more files to be removed in the *remove* field
29 2 Peter Amstutz
* rename: list the file with the old name in the *remove* field and the new name in the *add* field
30 2 Peter Amstutz
* modify: list the file with the old contents in the *remove* field and the new contents in the *add* field
31 2 Peter Amstutz
* combine collections: provide the full collection manifest in the *add* field