Arvados

Pipeline template development » History » Revision 2

Revision 1 (Bryan Cosca, 04/19/2016 03:30 PM) → Revision 2/15 (Bryan Cosca, 04/19/2016 07:55 PM)

h1. Pipeline template development 

 This wiki will describe how to write a pipeline template. Some documentation for writing a pipeline template using run-command is available on "doc.arvados.org":http://doc.arvados.org/user/tutorials/running-external-program.html 

 <pre> 
 "components": { 
  "JobName": { 
   "script": "JobScript", 
   "script_version": "master", 
   "repository": "yourname/yourname", 
   "script_parameters": { 
    "CollectionOne": { 
     "required": true, 
     "dataclass": "Collection" 
    }, 
    "ParameterOne":{ 
     "required": true, 
     "dataclass": "text", 
     "default": "ParameterOneString" 
    } 
   }, 
   "runtime_constraints": { 
    "docker_image": "bcosc/arv-base-java", 
    "arvados_sdk_version": "master" 
   } 
  } 
 } 
 </pre> 

 How to wrap a git repository containing a crunch script and a docker image into a component 
 Link to "Git Strategy for Pipeline Development" wiki page 

 h3. Writing How to write script_parameters 

 "Script_parameters":http://doc.arvados.org/api/schema/PipelineTemplate.html are inputs that can be called in your crunch script. Each script parameter can have any dataclass: Collection, File, number, text. Collection 
 dataclasses and their actual meanings ("Collection" passes in the pdh string (ex. 39c6f22d40001074f4200a72559ae7eb+5745), File passes in a file path in a collection (ex. 39c6f22d40001074f4200a72559ae7eb+5745/foo.txt), number passes in any integer, and text passes in any string. string, etc) 

 The default parameter is useful for using a collection you know will most likely be used, so the user does not have full "script_parameters documentation": http://doc.arvados.org/api/schema/PipelineTemplate.html 

 How to input it manually. For example, a reference genome collection that will be used throughout the entire pipeline. 

 The title and description parameters are useful for showing what the script parameter is doing, but is not necessary. 

 h3. Writing choose runtime_constraints 

 "Runtime_constraints":http://doc.arvados.org/api/schema/Job.html are inputs in your job that help choose node parameters that your pipeline will run on. Optimizing these parameters can be found in the "Pipeline_Optimization wiki.":https://dev.arvados.org/projects/arvados/wiki/Pipeline_Optimization 

 
 The actual meaning of min_nodes 
 Setting max_tasks_per_node != 1