Arvados

h1. Pipeline template development

This wiki will describe how to write a pipeline template. Some documentation for writing a pipeline template using run-command is already available on "doc.arvados.org.":http://doc.arvados.org/user/tutorials/running-external-program.html Here's an example pipeline template. More documentation for writing pipeline templates to run crunch scripts can be found "here.":https://dev.arvados.org/projects/arvados/wiki/Writing_a_Script_Calling_a_Third_Party_Tool

<pre>

"components": {

 "JobName": {

  "script": "JobScript.py",

  "script_version": "master",

  "repository": "yourname/yourname",

  "script_parameters": {

   "CollectionOne": {

    "required": true,

    "dataclass": "Collection"

   },

   "ParameterOne":{

    "required": true,

    "dataclass": "text",

    "default": "ParameterOneString"

   }

  },

  "runtime_constraints": {

   "docker_image": "bcosc/arv-base-java",

   "arvados_sdk_version": "master"

  }

 }

}

</pre>

The script used for the job is specified under the 'script' parameter, using the commit hash or branch name under 'script_version', which is under the arvados git repository specified under 'repository'. Note: Github repositories can also be used, as long as the repository is public. One important note is that your script must be in a folder called 'crunch_scripts'.

When developing a pipeline, we have an arvados best practices guideline for how to use your git repository effectively "here.":https://dev.arvados.org/projects/arvados/wiki/Git_strategy_for_pipeline_development

h3. Writing script_parameters

"Script_parameters":http://doc.arvados.org/api/schema/PipelineTemplate.html are inputs that can be called in your crunch script. Each script parameter can have any dataclass: Collection, File, number, text. Collection uses the pdh string (ex. 39c6f22d40001074f4200a72559ae7eb+5745), File passes in a file path in a collection (ex. 39c6f22d40001074f4200a72559ae7eb+5745/foo.txt), number passes in any integer, and text passes in any string.

The default parameter is useful for using a collection you know will most likely be used, so the user does not have to input it manually. For example, a reference genome collection that will be used throughout the entire pipeline.

The title and description parameters are useful for showing what the script parameter is doing, but is not necessary.

For example, pipeline template with script parameters:

<pre>

"reference_collection":{

 "required":true,

 "dataclass":"Collection"

},

"bwa_collection":{

 "required":true,

 "dataclass":"Collection",

 "default":"39c6f22d40001074f4200a72559ae7eb+5745"

},

 "sample":{

 "required":true,

 "dataclass":"Collection",

 "title":"Input FASTQ Collection",

 "description":"Input the fastq collection for BWA mem"

},

"read_group":{

 "required":true,

 "dataclass":"Text"

},

"additional_params":{

 "required":false,

 "dataclass":"Text"

},

</pre>

yields this view in workbench:

!7d1b807a78e3bd095d02913dd1074ddf.png!

The inputs tab in the pipeline instance page shows all the required parameters. You can click 'Choose' to grab a collection from a project for the reference_collection and input FASTQ Collection parameters. You can type in the read_group you want to use here as well. You can change the bwa_collection, but since you set the default collection, you only need to change it when you need to.

For the 'additional_params' parameter, since its not required, its in the 'Components' tab, where you can set it:

!752b7261b5710fbf362db26f315fc45d.png!

h3. Writing runtime_constraints

"Runtime_constraints":http://doc.arvados.org/api/schema/Job.html are inputs in your job that help choose node parameters that your pipeline will run on. Optimizing these parameters can be found in the "Pipeline_Optimization wiki.":https://dev.arvados.org/projects/arvados/wiki/Pipeline_Optimization

One runtime constraint is docker_image. It is suggested that while developing you use the latest version of the image, which you can specify by using the name of the image. When in production, you should use the portable data hash of the image you specifically want to use to avoid problems when accidentally changing the image or other conflicts.

Using min_nodes will spin up as many nodes as you've specified. Be warned that you can allocate your entire cluster to your job, so use this with caution.

The max_tasks_per_node parameter will allow you to allocate more computations on your node. By default, this is 1. If you are under utilizing your nodes, you can try to increase this number. Keep in mind that the total CPU/RAM/space usage of your tasks should fit on your node. It's very easy to overestimate the compute power of your tasks. Using something like "crunchstat-summary":https://dev.arvados.org/projects/arvados/wiki/Pipeline_Optimization should help bridge this gap.