Project

General

Profile

Actions

Bug #19382

open

Cannot stream stdin to arv-put with --update-collection

Added by Peter Amstutz about 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
SDKs
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

# curl http://www.google.com | arv-put --update-collection xxxxx-4zz18-m934edwa42cyvak --filename google_index_html  

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 14744    0 14744    0     0  95740      0 --:--:-- --:--:-- --:--:-- 98953

usage: arv-put [-h] [--version] [--normalize | --dry-run] [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw] [--update-collection UUID]

               [--use-filename FILENAME] [--filename FILENAME] [--portable-data-hash] [--replication N] [--storage-classes STORAGE_CLASSES] [--threads N] [--exclude PATTERN]

               [--follow-links | --no-follow-links] [--trash-at YYYY-MM-DDTHH:MM | --trash-after DAYS] [--project-uuid UUID] [--name NAME]

               [--progress | --no-progress | --batch-progress] [--silent] [--batch] [--resume | --no-resume] [--cache | --no-cache] [--retries RETRIES]

               [path ...]

arv-put: error: 

    --update-collection cannot be used when reading from stdin.

I think this was done to avoid dealing with the interaction between the "put cache" and the fact that stdin isn't a file -- I don't think there's any reason it can't work, we just chose to avoid the corner case by disallowing it.

A customer tried to do this and was confused that it didn't work. The use case is facilitating transfer from another storage system to Keep by piping to arv-put, where multiple files need to be placed into the collection. We should support it.

Actions #1

Updated by Peter Amstutz about 2 months ago

  • Description updated (diff)
Actions #2

Updated by Lucas Di Pentima about 2 months ago

Probably it's worth being explicit on how the collection's previous content would be overwritten by the new content, either with a new flag like --overwrite-collection or a log message (since now that we have versioned collections, is not so bad to overwrite a collection without notice)

Actions #3

Updated by Peter Amstutz about 2 months ago

I'm assuming it would add or replace (or even append to) the individual file, not change anything else in the collection. It might be good to have --overwrite or --append flags to indicate the desired behavior when there is a conflict with an existing file.

Actions #4

Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2022-09-14 sprint to 2022-09-28 sprint
Actions #5

Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2022-09-28 sprint to 2022-10-12 sprint
Actions #6

Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Actions

Also available in: Atom PDF