Cannot stream stdin to arv-put with --update-collection
# curl http://www.google.com | arv-put --update-collection xxxxx-4zz18-m934edwa42cyvak --filename google_index_html % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14744 0 14744 0 0 95740 0 --:--:-- --:--:-- --:--:-- 98953 usage: arv-put [-h] [--version] [--normalize | --dry-run] [--as-stream | --stream | --as-manifest | --in-manifest | --manifest | --as-raw | --raw] [--update-collection UUID] [--use-filename FILENAME] [--filename FILENAME] [--portable-data-hash] [--replication N] [--storage-classes STORAGE_CLASSES] [--threads N] [--exclude PATTERN] [--follow-links | --no-follow-links] [--trash-at YYYY-MM-DDTHH:MM | --trash-after DAYS] [--project-uuid UUID] [--name NAME] [--progress | --no-progress | --batch-progress] [--silent] [--batch] [--resume | --no-resume] [--cache | --no-cache] [--retries RETRIES] [path ...] arv-put: error: --update-collection cannot be used when reading from stdin.
I think this was done to avoid dealing with the interaction between the "put cache" and the fact that stdin isn't a file -- I don't think there's any reason it can't work, we just chose to avoid the corner case by disallowing it.
A customer tried to do this and was confused that it didn't work. The use case is facilitating transfer from another storage system to Keep by piping to arv-put, where multiple files need to be placed into the collection. We should support it.
Updated by Lucas Di Pentima about 2 months ago
Probably it's worth being explicit on how the collection's previous content would be overwritten by the new content, either with a new flag like
--overwrite-collection or a log message (since now that we have versioned collections, is not so bad to overwrite a collection without notice)
Updated by Peter Amstutz about 2 months ago
I'm assuming it would add or replace (or even append to) the individual file, not change anything else in the collection. It might be good to have --overwrite or --append flags to indicate the desired behavior when there is a conflict with an existing file.