Project

General

Profile

Actions
History

Arvados-cwl-runner development

Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow.

In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow, and gets information about the container it is running in using containers().current().

When doing development related to workflow execution, you can use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development "arvados/jobs" image with the correct code and version tag.

Example development process

Setup python virtual environment, edit code, and test locally

  1. $ python3 -m venv ./venv
  2. $ . venv/bin/activate
  3. $ (cd ~/arvados/sdk/cwl && pip install -e .)
  4. do your development
  5. $ arvados-cwl-runner --local

Commit code, build development docker jobs image, and test submission

  1. $ git commit
  2. $ (cd ~/arvados/sdk/cwl && pip install -e .)
  3. $ ~/arvados/build/build-dev-docker-jobs-image.sh
  4. $ arvados-cwl-runner --submit

Important to note:

  • You need to have ARVADOS_API_HOST and ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on Current token in the user dropdown menu.
  • "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp).
  • If you are using arvbox, the paths here refer to your development arvados repository in the .arvbox folder
  • You can specify custom schema salad and/or cwltool source to use in the dev docker jobs image by doing the following:
    • $ SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh
  • Enable all the debugging output with arvados-cwl-runner --debug
  • If you have a problem that can only be reproduced by running in the container environment, you can add log statements to a-c-r, cwltool and/or salad in order to get more insight as to what is happening. Be user to lower them to 'debug' level or remove them when you are done.

Testing

This section describes how to set up your testing environment and run/debug CWL conformance and Arvados integration tests. Tests can be run several different ways. You can run tests using a Python 3 virtualenv in a docker image running Python 2, vice versa, or in a pure Python 2 or 3 setup. Python 2 is the default for build-dev-docker-jobs-image.sh. Use "PYCMD=python3 arvados/build/build-dev-docker-jobs-image.sh" to get a Python 3 dev docker image. You can use the --python="path/to/python/exec/" virtualenv argument to specify your python version. You can also test both the jobs and containers APIs by changing the --api flag on run-tests.sh.

If using arvbox, you can more easily test with test_with_arvbox.sh, which is in the arvados/sdk/cwl folder. Commands include:

--suite (conformance|integration)

--api (jobs|containers)

Example: test_with_arvbox.sh --no-reset-container --leave-running --config dev --build --pythoncmd python3 --suite integration -j3

Environment setup

  1. $ "arvbox start" and "arvswitch" to your dev cluster or run "arvbox start dev"
  2. If ARVADOS_API_HOST and ARVADOS_API_TOKEN are not set, find the the workbench ip address from the arvbox output. Open it in a browser and add a new account with fake credentials. Click on Current token in the user dropdown menu and follow instructions. If you subsequently receive localhost match errors, use "export ARVADOS_API_HOST_INSECURE=1".
  3. $ virtualenv venv
  4. $ . venv/bin/activate
  5. $ (cd ~/arvados/sdk/cwl && pip install -e .)
  6. $ ~/arvados/build/build-dev-docker-jobs-image.sh.

Running CWL conformance tests

run_tests.sh is located in the common-workflow-language repository. To improve speed, use -jx, where x is the number tests to run simultaneously. To run a single test, use -nx where x is the test number zero-indexed.

  1. $ pip install cwltest
  2. $ git clone https://github.com/common-workflow-language/common-workflow-language.git

Run tests

  1. $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --local"
  2. $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --submit"

Debug tests

When a test fails, there should be output displaying Test failed: followed by the test command. Copy the command.

  1. $ cd common-workflow-language/v1.0
  2. $ Paste the command and replace the --quiet flag with --debug

Running arvados CWL integration tests

  1. $ cd arvados/sdk/cwl/tests
  2. $ ./arvados-tests.sh

Updated by Ward Vandewege about 3 years ago ยท 14 revisions