Pipeline template development » History » Version 2

Version 1 (Bryan Cosca, 04/19/2016 03:30 PM) → Version 2/15 (Bryan Cosca, 04/19/2016 07:55 PM)

h1. Pipeline template development

This wiki will describe how to write a pipeline template. Some documentation for writing a pipeline template using run-command is available on "doc.arvados.org":http://doc.arvados.org/user/tutorials/running-external-program.html

"components": {
"JobName": {
"script": "JobScript",
"script_version": "master",
"repository": "yourname/yourname",
"script_parameters": {
"CollectionOne": {
"required": true,
"dataclass": "Collection"
"required": true,
"dataclass": "text",
"default": "ParameterOneString"
"runtime_constraints": {
"docker_image": "bcosc/arv-base-java",
"arvados_sdk_version": "master"

How to wrap a git repository containing a crunch script and a docker image into a component
Link to "Git Strategy for Pipeline Development" wiki page

h3. Writing How to write script_parameters

"Script_parameters":http://doc.arvados.org/api/schema/PipelineTemplate.html are inputs that can be called in your crunch script. Each script parameter can have any dataclass: Collection, File, number, text. Collection

dataclasses and their actual meanings ("Collection"
passes in the pdh string (ex. 39c6f22d40001074f4200a72559ae7eb+5745), File passes in a file path in a collection (ex. 39c6f22d40001074f4200a72559ae7eb+5745/foo.txt), number passes in any integer, and text passes in any string. string, etc)

The default parameter is useful for using a collection you know will most likely be used, so the user does not have full "script_parameters documentation": http://doc.arvados.org/api/schema/PipelineTemplate.html

to input it manually. For example, a reference genome collection that will be used throughout the entire pipeline.

The title and description parameters are useful for showing what the script parameter is doing, but is not necessary.

h3. Writing
choose runtime_constraints

"Runtime_constraints":http://doc.arvados.org/api/schema/Job.html are inputs in your job that help choose node parameters that your pipeline will run on. Optimizing these parameters can be found in the "Pipeline_Optimization wiki.":https://dev.arvados.org/projects/arvados/wiki/Pipeline_Optimization

The actual meaning of min_nodes
Setting max_tasks_per_node != 1