Story #3015

Make gatk3 pipeline template

Added by Tom Clegg about 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
06/24/2014
Due date:
% Done:

80%

Estimated time:
(Total: 0.00 h)
Story points:
3.0

Description

  • Create a project
  • Download and add the appropriate reference & example datasets to the project
  • Make a docker image with all of the relevant redistributable software pre-installed
  • Make a "pirs" crunch script and use it to generate the simulation dataset based on hg19 chr1
  • Make a "Single sample SNV with bwa and gatk" pipeline with no parallel/asynchronous tasks
  • Time permitting, make another pipeline that splits the inputs as described below in order to get faster turnaround time when multiple nodes are available.
The attached script and the existing GATK exome pipeline should be helpful. Notes:
  • Use FUSE mount for inputs
  • GATK3 (like attached) not GATK2 (like existing pipeline)
  • Use a docker image with redistributable tools pre-installed, assuming this makes things easier (but not GATK itself - continue to pass this tarball as a job input)
  • Use the file-select script to get appropriate bits from the GATK bundle (which we should have an entire copy of in our project), rather than downloading individual files needed.
  • Existing pipeline provides clues (not necessarily all correct with latest tool versions) about which tools are capable of reading/writing pipes rather than regular files.

Notes about parallelizing:

We can split the FASTQ into many chunks as we want, however after the mapping, we should merge the alignments from one sample into single SAM/BAM file to stack the reads on each genome position. Then we split the SAM/BAM file again by chromosome. So roughly speaking we can get 24 or 25 BAM fragments then all downstream steps could be applied on these chromosome based BAM fragments. At last, probably after annotation, we merge fragment files into one final file. To increase parallelism we can even split the BAM on positions where have very low/no coverage.

Single_Sample_SNV_Pipeline.txt (6.82 KB) Single_Sample_SNV_Pipeline.txt Tom Clegg, 06/18/2014 01:04 AM

Subtasks

Task #3068: Write pipeline templateResolvedPeter Amstutz

Task #3107: Run on AWS instance (4xphq/qr1hi)ResolvedPeter Amstutz

Task #3066: Build docker imageResolvedPeter Amstutz

Task #3069: Run on test dataResolvedPeter Amstutz

Task #3067: Write scripts for each stageResolvedPeter Amstutz


Related issues

Related to Arvados - Bug #3373: [Sample pipelines] Improve SNV pipeline to accept example exome fastq data (2 pairs of reads) as a single input collection.Resolved07/30/2014

History

#1 Updated by Tom Clegg about 5 years ago

  • Target version set to 2014-07-16 Sprint

#3 Updated by Tom Clegg about 5 years ago

  • Description updated (diff)

#4 Updated by Peter Amstutz about 5 years ago

  • Assigned To set to Peter Amstutz

#5 Updated by Peter Amstutz about 5 years ago

Ongoing notes:

  • Can't upload with arv-put to 4xphq. Got "entity too large" error from nginx.
  • if I upload something with arv-put it doesn't go into any folders by default, and there's no link to the collections table any more

#6 Updated by Peter Amstutz about 5 years ago

  • No modal chooser for individual files. Workaround: select the files in a different window using the paperclip, then they show up in selection dropdown.
  • Pipelines created from pipeline templates don't get added to the same folder.
  • Job parameters passed to docker need to be JSON encoded
  • An "arv migrate" command for copying objects and collections between arvados instances seems like it would be a good idea.
  • if you try to queue a job while another one is running and it's running locally (not using slurm), it tries to run it immediately and then fails because the job directory is locked

#7 Updated by Tom Clegg almost 5 years ago

  • Status changed from New to In Progress

#8 Updated by Peter Amstutz almost 5 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF