Arvados-cwl-runner development » History » Revision 11
« Previous |
Revision 11/14
(diff)
| Next »
Eric Biagiotti, 02/05/2019 02:15 PM
Arvados-cwl-runner development¶
Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow.
In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow, and gets information about the container it is running in using containers().current().
When doing development related to workflow execution, you can use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development "arvados/jobs" image with the correct code and version tag.
Example development process¶
Setup python virtual environment, edit code, and test locally
- $ virtualenv venv
- $ . venv/bin/activate
- $ (cd ~/arvados/sdk/cwl && python setup.py develop)
- do your development
- $ arvados-cwl-runner --local
Commit code, build development docker jobs image, and test submission
- $ git commit
- $ (cd ~/arvados/sdk/cwl && python setup.py develop)
- $ ~/arvados/build/build-dev-docker-jobs-image.sh
- $ arvados-cwl-runner --submit
Important to note:
- You need to have ARVADOS_API_HOST and ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on Current token in the user dropdown menu.
- "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp).
- If you are using arvbox, the paths here refer to your development arvados repository in the .arvbox folder
- You can specify custom schema salad, cwltool packages, and/or the python executable to use in the dev docker jobs image by doing the following:
- $ PYCMD=python_or_python3 SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh
Testing¶
This section describes how to set up your testing environment and run/debug CWL conformance and Arvados integration tests. Tests can be run 4 different ways. You can run tests using a Python 3 virtualenv in a docker image running Python 2, vice versa, or in a pure Python 2 or 3 setup. Python 2 is the default for build-dev-docker-jobs-image.sh. Use "PYCMD=python3 arvados/build/build-dev-docker-jobs-image.sh" to get a Python 3 dev docker image. You can use the --python="path/to/python/exec/" virtualenv argument to specify your python version.
If using arvbox, you can more easily test with test_with_arvbox.sh, which is in the arvados/sdk/cwl folder. Commands include:
--pythoncmd (python|python3)
--suite (conformance|integration)
--api (jobs|containers)
Example: test_with_arvbox.sh --no-reset-container --leave-running --config dev --build --pythoncmd python3 --suite integration -j3
Environment setup¶
- $ "arvbox start" and "arvswitch" to your dev cluster or run "arvbox start dev"
- If ARVADOS_API_HOST and ARVADOS_API_TOKEN are not set, find the the workbench ip address from the arvbox output. Open it in a browser and add a new account with fake credentials. Click on Current token in the user dropdown menu and follow instructions. If you subsequently receive localhost match errors, use "export ARVADOS_API_HOST_INSECURE=1".
- $ virtualenv venv
- $ . venv/bin/activate
- $ (cd ~/arvados/sdk/cwl && python setup.py develop)
- $ ~/arvados/build/build-dev-docker-jobs-image.sh.
Running CWL conformance tests¶
run_tests.sh is located in the common-workflow-language repository. To improve speed, use -jx, where x is the number tests to run simultaneously. To run a single test, use -nx where x is the test number zero-indexed.
- $ pip install cwltest
- $ git clone https://github.com/common-workflow-language/common-workflow-language.git
Run tests
- $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --local"
- $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --submit"
Debug tests
When a test fails, there should be output displaying Test failed: followed by the test command. Copy the command.
- $ cd common-workflow-language/v1.0
- $ Paste the command and replace the --quiet flag with --debug
Running arvados CWL integration tests¶
- $ cd arvados/sdk/cwl/tests
- $ ./arvados-tests.sh
Updated by Eric Biagiotti over 5 years ago · 11 revisions