Pipeline Optimization¶

Crunchstat Summary¶

How to install crunchstat-summary¶

How to use crunchstat-summary¶

--text mode
using node recommendations, keep cache size

--html mode
check if you're cpu/io bound
check if tasks are being weird, i.e. gatk queue case

when to pipe and when to write to keep
in general writing straight to keep will reap benefits. If you run crunchstat-summary --html and you see keep io stopping once in a while, then youre cpu bound.

How to optimize the number when you don't have native multithreading¶

tools like gatk, blah blah have native multithreading where you pass a -t.
tools like varscan/freebayes blah blah don't have native multithreading so you need to find a workaround. generally, some tools have a -L --intervals to pass in certain loci to work on. If you have a bed file you can split on, then you can create a new task per interval.
example here

piping between tools or writing to a tmpdir.¶

choosing the right number of jobs¶

each job must output a collection, so if you don't want to output a file, then