Feature #10191
open
Arvados should be able to cat/zcat files in collections easily
Added by Bryan Cosca about 8 years ago.
Updated 10 months ago.
Release relationship:
Auto
Description
Use case:
I have 8 lanes of paired end fastq files, and I would like to combine all the lanes together into one big sample. I don't want to manually cat them, as that could take upwards of 12 hours.
While server-side concatenation could be valuable in certain circumstances, experiments for this case showed that the actual time was 13 min., not 12 hours, so we're going to defer looking at this for the time being.
Support for server-side (manifest-based) concatenation seems like a potentially useful feature to have, although I'm wary of the `zcat` support unless it is also just an underlying manifest concatenation. If non-trivial operations such (re)compression need to be exposed via the FUSE interface, I'd suggest the use of more generic support for running pipelines via keep such as that proposed in story 10388.
- Tracker changed from Bug to Feature
- Target version set to Arvados Future Sprints
- Target version deleted (
Arvados Future Sprints)
- Target version set to Future
Also available in: Atom
PDF
Updated by Tom Morris 
Updated by 
Updated by 
Updated by