Arvados

h1. Git strategy for pipeline development

The following scenario is common:

* You have a project that involves one or two pipelines

* Each pipeline have many components

* These pipelines and components make use of a common code base

Example:

|crunch scripts in repo|pipeline A|pipeline B|

|<pre>crunch_scripts/align

crunch_scripts/call

crunch_scripts/compare</pre>|<pre>align(1)    align(2)

   |           |

call(1)     call(2)

   |____   ____|

        | |

      compare</pre>|<pre>align(1)    align(2)    align(3)

   |           |           |

call(1)     call(2)     call(3)

   |_________  |  _________|

             | | |

            compare</pre>|

While developing the code you can expect to have moments like these:

* Fix a bug in @compare@ that was making it fail when given 3 inputs.

* Update the code, commit, push, and re-run. (Note: results from previous runs _that succeeded_ are still valid.)

* Find a bug in @compare@ that was making it produce incorrect output when given 3 inputs.

* Update the code, commit, push, and re-run. Update pipeline template B to prevent the broken jobs (the ones that are marked "success" but produced incorrect outputs) from being re-used in future pipeline runs. (Note: results from previous jobs from pipeline A are still OK.)

h2. A strategy