Arvados

h1. Pipeline Optimization

h2. Crunchstat Summary

h3. How to install crunchstat-summary

h3. How to use crunchstat-summary

--text mode

using node recommendations, keep cache size

--html mode

check if you're cpu/io bound

check if tasks are being weird, i.e. gatk queue case

when to pipe and when to write to keep

in general writing straight to keep will reap benefits. If you run crunchstat-summary --html and you see keep io stopping once in a while, then youre cpu bound.

h3. How to optimize the number when you don't have native multithreading

tools like gatk, blah blah have native multithreading where you pass a -t.

tools like varscan/freebayes blah blah don't have native multithreading so you need to find a workaround. generally, some tools have a -L --intervals to pass in certain loci to work on. If you have a bed file you can split on, then you can create a new task per interval.

example here

h3. piping between tools or writing to a tmpdir.

h3. choosing the right number of jobs

each job must output a collection, so if you don't want to output a file, then