Idea #12060
Updated by Peter Amstutz over 7 years ago
Goal: have arvados-cwl-runner execute CWL files from a git branch as written, avoiding the "packing" step.
Execution:
# crunch-run can mount git repositories (#8311)
# arvados-cwl-runner, running inside container, uploads file references to Keep and updates internal workflow references to maintain job reuse.
Publishing:
# workflow record gets new fields: @git_repository_uuid, git_commit, workflow_path, properties@ (for metadata)
# "publish" API call takes git_repository_uuid, git_commit, workflow_path; scans workflow, checks validity, and creates/updates workflow record with extracted metadata (such as name, description, inputs)
# "Run" button in workbench (or API call?) sets up container with mount point: git_repository_uuid, git_commit; runs arvados-cwl-runner with CWL file located at git mount/workflow_path
Alternative to explicit "publish" API is to use a git hook on commit which scans repository for CWL files matching a certain pattern and registers/updates them automatically. This closes the loop and make publishing a workflow as simple as a git commit. More discussion/proof of concept at https://dev.arvados.org/issues/10344#note-4
Related:
The cwlviewer has a scheme for permalinks to CWL files hosted in git:
https://github.com/common-workflow-language/cwlviewer/wiki/Permalinks