Project

General

Profile

Pipeline Optimization » History » Revision 4

Revision 3 (Bryan Cosca, 04/14/2016 06:03 PM) → Revision 4/31 (Bryan Cosca, 04/14/2016 06:16 PM)

h1. Pipeline Optimization 

 h2. Crunchstat Summary 

 Crunchstat-summary is an arvados tool to help choose optimal configurations for arvados jobs and pipeline instances. It helps you choose "runtime_constraints":http://doc.arvados.org/api/schema/Job.html specified in the pipeline template under each job, as well as graph general statistics for the job, for example, CPU usage, RAM, and Keep network traffic across the duration of a job. 

 h3. How to install crunchstat-summary 

 <pre> 
 $ git clone https://github.com/curoverse/arvados.git 
 $ cd arvados/tools/crunchstat-summary/ 
 $ python setup.py build 
 $ python setup.py install --user 
 </pre> 

 h3. How to use crunchstat-summary 

 <pre> 
 $ ./bin/crunchstat-summary --help 
 usage: crunchstat-summary [-h] 
                           [--job UUID | --pipeline-instance UUID | --log-file LOG_FILE] 
                           [--skip-child-jobs] [--format {html,text}] 
                           [--verbose] 

 Summarize resource usage of an Arvados Crunch job 

 optional arguments: 
   -h, --help              show this help message and exit 
   --job UUID              Look up the specified job and read its log data from 
                         Keep (or from the Arvados event log, if the job is 
                         still running) 
   --pipeline-instance UUID 
                         Summarize each component of the given pipeline 
                         instance 
   --log-file LOG_FILE     Read log data from a regular file 
   --skip-child-jobs       Do not include stats from child jobs 
   --format {html,text}    Report format 
   --verbose, -v           Log more information (once for progress, twice for 
                         debug) 
 </pre> 

 --text mode 
 using node recommendations, keep cache size 

 --html mode 
 check if you're cpu/io bound 
 check if tasks are being weird, i.e. gatk queue case 

 when to pipe and when to write to keep 
 in general writing straight to keep will reap benefits. If you run crunchstat-summary --html and you see keep io stopping once in a while, then youre cpu bound. If you're seeing cpu level off and keep-read or keep-write taking too long, then you're io bound. 

 h3. choosing the right number of jobs 

 each job must output a collection, so if you don't want to output a file, then  

 h2. Job Optimization 
 h3. How to optimize the number when you don't have native multithreading 

 tools like gatk, blah blah have native multithreading where you pass a -t. 
 tools like varscan/freebayes blah blah don't have native multithreading so you need to find a workaround. generally, some tools have a -L --intervals to pass in certain loci to work on. If you have a bed file you can split on, then you can create a new task per interval. 
 example here 

 h3. piping between tools or writing to a tmpdir.