Arvados-cwl-runner development¶

Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow.

In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow, and gets information about the container it is running in using containers().current().

When doing development related to workflow execution, you can use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development "arvados/jobs" image with the correct code and version tag.

Example development process¶

Setup python virtual environment, edit code, and test locally

1. $ virtualenv venv
2. $ . venv/bin/activate
3. $ (cd ~/arvados/sdk/cwl && python setup.py develop)
4. do your development
5. $ arvados-cwl-runner --local

Commit code, build development docker jobs image, and test submission

1. $ git commit
2. $ (cd ~/arvados/sdk/cwl && python setup.py develop)
3. $ ~/arvados/build/build-dev-docker-jobs-image.sh
4. $ arvados-cwl-runner --submit

Important to note:

• You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on Current token in the user dropdown menu.
• "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp).
• If you are using arvbox, the paths above refer to your development arvados repository in the .arvbox folder
• You can specify custom schema salad, cwltool packages, and/or the python executable to use in the dev docker jobs image by doing the following:
  • $ PYCMD=python_or_python3 SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh

Running cwl conformance tests¶

1. $ "arvbox start" and "arvswitch" to your dev cluster
2. $ virtualenv venv
3. $ . venv/bin/activate
4. $ (cd ~/arvados/sdk/cwl && python setup.py develop)
5. $ pip install cwltest
6. $ git clone https://github.com/common-workflow-language/common-workflow-language.git
7. $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse"

Running arvados integration tests¶

1. $ virtualenv venv
2. $ . venv/bin/activate
3. $ (cd ~/arvados/sdk/cwl && python setup.py develop)
4. $ cd arvados/sdk/cwl/tests
5. $ ./arvados-tests.sh