Crunch » History » Revision 16

Revision 15 (Ward Vandewege, 11/06/2017 07:44 PM) → Revision 16/21 (Ward Vandewege, 11/06/2017 07:45 PM)

h1. Crunch - container orchestration 

 Arvados has a robust container orchestration system called 'Crunch', which executes CWL workflows while maintaining provenance and reproducibility.  

 h2. Design Goals 

 Notable design goals and features include: 

 * Make use of multiple cores and nodes to produce results faster 
 * Integrate with [[Keep]] and git repositories to maintain provenance 
 * Use off-the-shelf software tools in distributed computations 
 * Efficient over a wide range of problem sizes 
 * Maximum flexibility of programming language choice 
 * Maximum flexibility of execution environment 
 * Tools for building reusable pipelines 
 * Lower entry barrier for users 


 h2. Benefits of Crunch 

 Although some of the workflow and provenance features in Arvados could theoretically be implemented using Hadoop MapReduce, there are distinct benefits to Crunch: 

 * *Provenance and Reproducibility* - Like Keep, the Arvados distributed file system, Crunch is designed to automatically track automate tracking the origin of result data. It can also efficiently reproduce data, reproducing complex workflows pipelines, and comparing workflows pipelines to one another. 

 * *Performance* - Most genomics problems are embarrassingly parallel and can benefit from horizontal scaling. In the cloud, Crunch can deliver cost-effective performance for genomics related analyses by automatically adjusting the available compute resources to the workload. 

 * *Standardization* - "Common Workflow Language (CWL)": is the workflow description standard in bioinformatics. It is the native workflow language in Crunch.