Project

General

Profile

Idea #12060

Updated by Peter Amstutz over 6 years ago

Goal: have arvados-cwl-runner execute CWL files from a git branch as written, avoiding the "packing" step. 

 Execution: 

 # crunch-run can mount git repositories (#8311) 
 # arvados-cwl-runner, running inside container, uploads file references to Keep and updates internal workflow references to maintain job reuse. 

 Publishing: 

 # workflow record gets new fields: @git_repository_uuid, git_commit, workflow_path, properties@ (for metadata) 
 # "publish" API call takes git_repository_uuid, git_commit, workflow_path; scans workflow, checks validity, and creates/updates workflow record with extracted metadata (such as name, description, inputs) 
 # "Run" button in workbench (or API call?) sets up container with mount point: git_repository_uuid, git_commit; runs arvados-cwl-runner with CWL file located at git mount/workflow_path 

 Alternative to explicit "publish" API is to use a API: git hook on commit which scans repository for CWL files matching a certain pattern and registers/updates them automatically.    This closes the loop and make publishing a workflow as simple as a git commit.    More discussion/proof automatically (proof of concept at https://dev.arvados.org/issues/10344#note-4 https://dev.arvados.org/issues/10344#note-8) 

Back