Build docker images as part of a workflow¶

(draft)

Background

Container images provide a well-defined execution environment for doing reproducible work. As long as the image is runnable by a container engine, a job can be repeated. However, the point of reproducibility isn't just to allow repetition of the same computation -- it's to make it possible to use prior work as the starting point for future work. Much of this opportunity is lost if the provenance trail ends at a giant binary image.

Example: When a bug is discovered in an analysis tool or library, it should be tractable to identify which existing results are affected and re-run those analyses with the updated software.

Requirements

Users should have the option of building container images...

• ...as part of a CWL workflow (so they can update the image-building instructions and hit one "re-run" button to see the result)
• ...in Arvados containers (so the build environment is controlled, build logs are saved, etc.)
• ...without having docker on the client side (so build-and-run workflows can be initiated from browsers, non-Linux workstations, and shared VM environments)