Bug #8910

[SDK] arv-put should save manifest text on API error

Added by Peter Amstutz about 5 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

If arv-put gets an API error when creating the collection record, the entire upload is for nothing because the once the manifest_text is lost it is necessary to re-upload the whole thing. Arv-put should save the manifest text somewhere to offer some way of recovering from error.

This should probably leverage the existing "resume" functionality.


Related issues

Related to Arvados - Bug #8878: Keep: sudden appearance of "missing" blocksClosed04/04/2016

Blocked by Arvados - Story #9463: [SDKs] Change arv-put to use the Collection class under the hoodResolved07/11/2016

Associated revisions

Revision 67c0bb9a (diff)
Added by Lucas Di Pentima almost 5 years ago

9463: Finished first draft on arv-put command use of Collection class. Also, partial uploads resuming is working. Tests were written to validate this. Refs #9463 #8910

History

#1 Updated by Peter Amstutz about 5 years ago

  • Description updated (diff)

#2 Updated by Brett Smith about 5 years ago

Peter Amstutz wrote:

This should probably leverage the existing "resume" functionality.

Another approach would be for arv-put to start using the Collection class, start a new collection on upload, and periodically save it until the upload is finished.

This is a bigger change, but moving to the Collection class makes it easier to make a lot of other changes we want in arv-put: refreshing permission tokens as it goes along, smarter block splits, adding files to existing collections, etc.

#3 Updated by Brett Smith about 5 years ago

  • Target version set to Arvados Future Sprints

#4 Updated by Tom Morris over 4 years ago

  • Assigned To set to Lucas Di Pentima

Lucas - have the recent arv-put changes resolved this?

#5 Updated by Lucas Di Pentima over 4 years ago

Tom Morris wrote:

Lucas - have the recent arv-put changes resolved this?

Now arv-put periodically (every 20 secs, IIRC) saves the manifest on its cache, so I would say: yes.

#6 Updated by Lucas Di Pentima about 4 years ago

  • Status changed from New to Resolved

#7 Updated by Tom Clegg about 4 years ago

  • Target version deleted (Arvados Future Sprints)

Also available in: Atom PDF