Feature #10191

Arvados should be able to cat/zcat files in collections easily

Added by Bryan Cosca almost 3 years ago. Updated over 2 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
10/05/2016
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Use case:

I have 8 lanes of paired end fastq files, and I would like to combine all the lanes together into one big sample. I don't want to manually cat them, as that could take upwards of 12 hours.

History

#1 Updated by Tom Morris almost 3 years ago

While server-side concatenation could be valuable in certain circumstances, experiments for this case showed that the actual time was 13 min., not 12 hours, so we're going to defer looking at this for the time being.

#2 Updated by Joshua Randall over 2 years ago

Support for server-side (manifest-based) concatenation seems like a potentially useful feature to have, although I'm wary of the `zcat` support unless it is also just an underlying manifest concatenation. If non-trivial operations such (re)compression need to be exposed via the FUSE interface, I'd suggest the use of more generic support for running pipelines via keep such as that proposed in story 10388.

#3 Updated by Tom Morris over 2 years ago

  • Tracker changed from Bug to Feature
  • Target version set to Arvados Future Sprints

Also available in: Atom PDF