Pipeline template development¶

This wiki will describe how to write a pipeline template. Some documentation for writing a pipeline template using run-command is available on doc.arvados.org

"components": {
 "JobName": {
  "script": "JobScript",
  "script_version": "master",
  "repository": "yourname/yourname",
  "script_parameters": {
   "CollectionOne": {
    "required": true,
    "dataclass": "Collection" 
   },
   "ParameterOne":{
    "required": true,
    "dataclass": "text",
    "default": "ParameterOneString" 
   }
  },
  "runtime_constraints": {
   "docker_image": "bcosc/arv-base-java",
   "arvados_sdk_version": "master" 
  }
 }
}

How to wrap a git repository containing a crunch script and a docker image into a component
Link to "Git Strategy for Pipeline Development" wiki page

Writing script_parameters¶

Script_parameters are inputs that can be called in your crunch script. Each script parameter can have any dataclass: Collection, File, number, text. Collection passes in the pdh string (ex. 39c6f22d40001074f4200a72559ae7eb+5745), File passes in a file path in a collection (ex. 39c6f22d40001074f4200a72559ae7eb+5745/foo.txt), number passes in any integer, and text passes in any string.

The default parameter is useful for using a collection you know will most likely be used, so the user does not have to input it manually. For example, a reference genome collection that will be used throughout the entire pipeline.

The title and description parameters are useful for showing what the script parameter is doing, but is not necessary.

Writing runtime_constraints¶

Runtime_constraints are inputs in your job that help choose node parameters that your pipeline will run on. Optimizing these parameters can be found in the Pipeline_Optimization wiki.

The actual meaning of min_nodes
Setting max_tasks_per_node != 1