Arvados-cwl-runner development » History » Revision 5
« Previous |
Revision 5/14
(diff)
| Next »
Peter Amstutz, 08/10/2018 03:36 PM
Arvados-cwl-runner development¶
Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow
In submit mode (--submit) it creates a container request which executes arvados-cwl-runner inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow (--local), and gets information about the container it is running in using containers().current().
When doing development related to workflow execution, you can also use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development arvados/jobs image with the correct code and version tag.
Here's a summary of the development process:
- $ virtualenv venv
- $ . venv/bin/activate
- $ (cd ~/arvados/sdk/cwl && python setup.py develop)
- do your development
- test locally
- $ arvados-cwl-runner --local
- $ git commit
- $ (cd ~/arvados/sdk/cwl && python setup.py develop)
- $ ~/arvados/build/build-dev-docker-jobs-image.sh
- test submit to cluster
- $ arvados-cwl-runner --submit
Important to note:
- You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment
- "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp).
- the a-c-r code which is run by the submitter and which is copied in the docker image are expected to match, and it uses the version number of the python package / docker image to enforce that.
Testing changes to Arvados Python SDK, cwltool or schema-salad¶
If changes
Running cwl conformance tests¶
- $ virtualenv venv
- $ . venv/bin/activate
- $ (cd ~/arvados/sdk/cwl && python setup.py develop)
- $ pip install cwltest
- $ git clone https://github.com/common-workflow-language/common-workflow-language.git
- $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse"
Running arvados integration tests¶
- $ virtualenv venv
- $ . venv/bin/activate
- $ (cd ~/arvados/sdk/cwl && python setup.py develop)
- $ cd arvados/sdk/cwl/tests
- $ ./arvados-tests.sh
Updated by Peter Amstutz about 6 years ago · 5 revisions