Collection update API¶

Problem: adding, updating, renaming, or removing a file in a collection requires sending the entire manifest text of the collection. This approach creates two problems:

1. It is inefficient; resources are wasted transferring and processing manifest data that hasn't changed
2. It creates a race condition; for example processes A and B read the collection record at time T1, merge it with their local updates, process A sends its update at time T2, process B sends its update at time T3; this causes process A's updates to be dropped.

Proposed solution¶

Introduce new 'update' or 'patch' endpoint for collections supporting a patching protocol allowing incremental collection update.

The protocol consists of two fields: remove and add. Each field contains a fragment of manifest text listing one or more files to be operated on.

The remove field lists files that should be removed from the collection. Remove is processed before add.
For each file, check that the file in the collection has the same content (== same blocks/segments) as the file to be removed. If the file contents match, the file is removed. If the file in tho collection file does not match the file to be removed, the file in the collection is renamed to a new unique name that indicates it is in conflict. If the file is not present in the collection, do nothing.

The add field lists files that should be added to the manifest. If a file does not exist in the collection, it is added. If the file already exists in the collection, the existing file in the collection is renamed to a new unique name that indicates it is in conflict; the new file is put in its place.

After the patch call, respond with the portable data hash. The caller can compare the PDH to the expected result to determine if there were conflicts that require it to update its local collection contents.

Any other files in the collection that are not mentioned in the remove or add operations are unaffected.

Use¶

With this protocol we can implement add, delete, rename, modify, and combine collections:

• add: provide one or more files to be added in the add field
• remove: provide one or more files to be removed in the remove field
• rename: list the file with the old name in the remove field and the new name in the add field
• modify: list the file with the old contents in the remove field and the new contents in the add field
• combine collections: provide the full collection manifest in the add field