Project

General

Profile

Arvados-cwl-runner development » History » Version 9

Eric Biagiotti, 01/28/2019 04:41 PM

1 1 Peter Amstutz
h1. Arvados-cwl-runner development
2
3 9 Eric Biagiotti
Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow.
4 2 Peter Amstutz
5 9 Eric Biagiotti
In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container.  The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance.  The a-c-r instance inside the container actively manages the workflow, and gets information about the container it is running in using containers().current().
6 2 Peter Amstutz
7 9 Eric Biagiotti
When doing development related to workflow execution, you can use "--local" to run a locally managed workflow using your development code.  However, you still need to test that the workflow can be submitted and run in the container.  This requires building a development "arvados/jobs" image with the correct code and version tag.
8 1 Peter Amstutz
9 9 Eric Biagiotti
h2. Example development process
10 2 Peter Amstutz
11 9 Eric Biagiotti
_Setup python virtual environment, edit code, and test locally_
12
13 1 Peter Amstutz
# $ virtualenv venv
14
# $ . venv/bin/activate
15
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
16 2 Peter Amstutz
# _do your development_
17
# $ arvados-cwl-runner --local
18 9 Eric Biagiotti
19
_Commit code, build development docker jobs image, and test submission_
20
21 2 Peter Amstutz
# $ git commit
22
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
23
# $ ~/arvados/build/build-dev-docker-jobs-image.sh
24
# $ arvados-cwl-runner --submit
25 3 Peter Amstutz
26 9 Eric Biagiotti
_Important to note:_
27 6 Peter Amstutz
28 9 Eric Biagiotti
* You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on _Current token_ in the user dropdown menu. 
29 7 Peter Amstutz
* "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp).
30 9 Eric Biagiotti
* If you are using arvbox, the paths above refer to your development arvados repository in the .arvbox folder
31
* You can specify custom schema salad, cwltool packages, and/or the python executable to use in the dev docker jobs image by doing the following:
32
    
33
     * $ PYCMD=python_or_python3 SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh
34 5 Peter Amstutz
35 4 Peter Amstutz
h2. Running cwl conformance tests
36
37 8 Peter Amstutz
# $ "arvbox start" and "arvswitch" to your dev cluster  
38 4 Peter Amstutz
# $ virtualenv venv
39
# $ . venv/bin/activate
40
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
41
# $ pip install cwltest
42
# $ git clone https://github.com/common-workflow-language/common-workflow-language.git
43
# $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse"
44
45
h2. Running arvados integration tests
46
47
# $ virtualenv venv
48
# $ . venv/bin/activate
49
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
50
# $ cd arvados/sdk/cwl/tests
51
# $ ./arvados-tests.sh