Project

General

Profile

Actions

Git strategy for pipeline development » History » Revision 1

Revision 1/2 | Next »
Tom Clegg, 02/16/2016 07:00 PM


Git strategy for pipeline development

The following scenario is common:
  • You have a project that involves one or two pipelines
  • Each pipeline have many components
  • These pipelines and components make use of a common code base

Example:

crunch scripts in repo pipeline A pipeline B
crunch_scripts/align
crunch_scripts/call
crunch_scripts/compare
align(1)    align(2)
   |           |
call(1)     call(2)
   |____   ____|
        | |
      compare
align(1)    align(2)    align(3)
   |           |           |
call(1)     call(2)     call(3)
   |_________  |  _________|
             | | |
            compare
While developing the code you can expect to have moments like these:
  • Fix a bug in compare that was making it fail when given 3 inputs.
  • Update the code, commit, push, and re-run. (Note: results from previous runs that succeeded are still valid.)
  • Find a bug in compare that was making it produce incorrect output when given 3 inputs.
  • Update the code, commit, push, and re-run. Update pipeline template B to prevent the broken jobs (the ones that are marked "success" but produced incorrect outputs) from being re-used in future pipeline runs. (Note: results from previous jobs from pipeline A are still OK.)

A strategy

Updated by Tom Clegg about 8 years ago · 1 revisions