Idea #12060
Updated by Peter Amstutz over 6 years ago
Goal: have arvados-cwl-runner execute CWL files from a git branch as written, avoiding the "packing" step. Execution: # crunch-run can mount git repositories (#8311) # arvados-cwl-runner, running inside container, uploads file references to Keep and updates internal workflow references to maintain job reuse. UI: Publishing: # User loads the CWL file in Composer workflow record gets new fields: @git_repository_uuid, git_commit, workflow_path, properties@ (for metadata) # Pushes a "publish" API call takes git_repository_uuid, git_commit, workflow_path; scans workflow, checks validity, and creates/updates workflow record with extracted metadata (such as name, description, inputs) # "Run" button # Composer creates a container request ## Puts minimal input description in "workflow.json" where workbench knows to look for it ## Put real workflow at a different (or API call?) sets up container with mount point point: git_repository_uuid, git_commit; runs arvados-cwl-runner with reference CWL file located at git mount/workflow_path Alternative to explicit "publish" API is to use a git repo & hook on commit ## Command line references real which scans repository for CWL files matching a certain pattern and registers/updates them automatically. This closes the loop and make publishing a workflow # Composer opens workbench page as simple as a git commit. More discussion/proof of concept at https://dev.arvados.org/issues/10344#note-4 Related: The cwlviewer has a scheme for container request permalinks to CWL files hosted in a new tab git: https://github.com/common-workflow-language/cwlviewer/wiki/Permalinks