Arvados

h1. Collection update API

Problem: adding, updating, renaming, or removing a file in a collection requires sending the entire manifest text of the collection.  This approach creates two problems:

# It is inefficient; resources are wasted transferring and processing manifest data that hasn't changed 

# It creates a race condition; for example processes A and B read the collection record at time T1, merge it with their local updates, process A sends its update at time T2, process B sends its update at time T3; this causes process A's updates to be dropped.

Proposed solution: introduce new 'update' or 'patch' endpoint for collections supporting a patching protocol allowing incremental collection update.

The protocol consists of two fields: *remove* and *add*.  Each field contains a fragment of manifest text.

The *remove* field lists files that should be removed from the collection.  Remove is processed before add.

For each file, check that the file in the collection has the same content (== same blocks/segments) as the file to be removed.  If the file contents match, the file is removed.  If the file in tho collection file does not match the file to be removed, the file in the collection is renamed to a new unique name that indicates it is in conflict.  If the file is not present in the collection, do nothing.

The *add* field lists files that should be added to the manifest.  If a file does not exist in the collection, it is added.  If the file  already exists in the collection, the existing file in the collection is renamed to a new unique name that indicates it is in conflict; the new file is put in its place.

After the patch call, respond with the portable data hash.  The caller can compare the PDH to the expected result to determine if there were conflicts that require it to update its local collection contents.

Any other files in the collection that are not mentioned in the *remove* or *add* operations are unaffected.

With this protocol we can implement add, delete, rename, modify, and combine collections:

* add: provide one or more files to be added in the *add* field

* remove: provide one or more files to be removed in the *remove* field

* rename: list the file with the old name in the *remove* field and the new name in the *add* field

* modify: list the file with the old contents in the *remove* field and the new contents in the *add* field

* combine collections: provide the full collection manifest in the *add* field