Arvados-cwl-runner development » History » Revision 9
Revision 8 (Peter Amstutz, 01/11/2019 09:37 PM) → Revision 9/14 (Eric Biagiotti, 01/28/2019 04:41 PM)
h1. Arvados-cwl-runner development Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow. workflow In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow, workflow (--local), and gets information about the container it is running in using containers().current(). When doing development related to workflow execution, you can also use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development "arvados/jobs" arvados/jobs image with the correct code and version tag. h2. Example Here's a summary of the development process process: _Setup python virtual environment, edit code, and test locally_ # $ virtualenv venv # $ . venv/bin/activate # $ (cd ~/arvados/sdk/cwl && python setup.py develop) # _do your development_ # _test locally_ # $ arvados-cwl-runner --local _Commit code, build development docker jobs image, and test submission_ # $ git commit # $ (cd ~/arvados/sdk/cwl && python setup.py develop) # $ ~/arvados/build/build-dev-docker-jobs-image.sh # _test submit to cluster_ # $ arvados-cwl-runner --submit _Important Important to note:_ note: * You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on _Current token_ in the user dropdown menu. environment * "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp). * If you the a-c-r code which is run by the submitter and which is copied in the docker image are using arvbox, expected to match, and it uses the paths above refer version number of the python package / docker image to your development arvados repository in enforce that. h2. Building a jobs image with custom cwltool or schema-salad When you build a "dev" jobs image, it generates custom arvados-python-client and arvados-cwl-runner packages from the .arvbox folder * working directory and injects them into the docker image build. You can also specify custom schema salad, salad and/or cwltool packages, and/or the python executable to use in the dev docker jobs image by doing the following: * packages this way: $ PYCMD=python_or_python3 SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh h2. Running cwl conformance tests # $ "arvbox start" and "arvswitch" to your dev cluster # $ virtualenv venv # $ . venv/bin/activate # $ (cd ~/arvados/sdk/cwl && python setup.py develop) # $ pip install cwltest # $ git clone https://github.com/common-workflow-language/common-workflow-language.git # $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse" h2. Running arvados integration tests # $ virtualenv venv # $ . venv/bin/activate # $ (cd ~/arvados/sdk/cwl && python setup.py develop) # $ cd arvados/sdk/cwl/tests # $ ./arvados-tests.sh