Feature #9136

[Python SDK] Add a Collection method to efficiently concatenate files

Added by Sarah Guthrie about 3 years ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
To Be Groomed
Start date:
05/04/2016
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Concatenating N files together to create a new one can be accomplished with manifest text manipulation, without moving data.

  1. Add a method to the Arvados SDK that appends the segments of one ArvadosFile to another (a generalized version could replace a region of one ArvadosFile with a region from another one.)
  2. Provide example Python code (create a new ArvadosFile that is the combinations of two or more other files)
  3. Provide example CWL (requires passing in explicit keep references, talking to API server to create new output collection, and producing cwl.output.json with references to the new collection).

History

#1 Updated by Brett Smith about 3 years ago

  • Subject changed from [Python SDK] Add an ability to 'cat' multiple files in a collection together to [Python SDK] Add a Collection method to efficiently concatenate files

#2 Updated by Brett Smith about 3 years ago

  • Target version set to Arvados Future Sprints

#3 Updated by Tom Morris 5 months ago

  • Target version changed from Arvados Future Sprints to To Be Groomed

#4 Updated by Peter Amstutz 5 months ago

  • Description updated (diff)

Also available in: Atom PDF