Project

General

Profile

Actions

Idea #10383

closed

Support incremental upload (rsync style) for arv-put

Added by Tom Morris over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
0.5

Description

Be able to redo an upload of a directory which has changed and only upload the new and changed files.

Functionality:

  • If a local file doesn't exist in the remote collection, upload it.
  • If a local file is different from a corresponding remote file, replace the remote collection with the local file contents.
    • As a special case, if a local file has been partially uploaded, try to resume upload and append remote file contents instead of starting from 0.
  • If a file exists remotely and has been deleted locally, the remote file is left unchanged (left for future feature).

In "resume" mode, the data is uploaded to Keep and checkpointed to cache. When upload is complete, a new collection is created.

In "update" mode, the data is uploaded to Keep and checkpointed to both the cache and the remote collection.

In both modes, read and record a checkpoint file which stores metadata about files that have been partially or completely uploaded.

When performing a resume or update, compare checkpointed file metadata to the remote collection and skip files which are unchanged.

Use ArvFile.manifest_text() to get a valid manifest for each individual file for the purposes of comparing content.


Subtasks 1 (0 open1 closed)

Task #10384: Review 10383-arv-put-incremental-uploadResolvedPeter Amstutz10/26/2016Actions
Actions

Also available in: Atom PDF