Project

General

Profile

Arvados-cwl-runner development » History » Version 13

Peter Amstutz, 10/13/2021 08:25 PM

1 1 Peter Amstutz
h1. Arvados-cwl-runner development
2
3 9 Eric Biagiotti
Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow.
4 2 Peter Amstutz
5 9 Eric Biagiotti
In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container.  The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance.  The a-c-r instance inside the container actively manages the workflow, and gets information about the container it is running in using containers().current().
6 2 Peter Amstutz
7 9 Eric Biagiotti
When doing development related to workflow execution, you can use "--local" to run a locally managed workflow using your development code.  However, you still need to test that the workflow can be submitted and run in the container.  This requires building a development "arvados/jobs" image with the correct code and version tag.
8 1 Peter Amstutz
9 9 Eric Biagiotti
h2. Example development process
10 2 Peter Amstutz
11 9 Eric Biagiotti
_Setup python virtual environment, edit code, and test locally_
12
13 1 Peter Amstutz
# $ virtualenv venv
14
# $ . venv/bin/activate
15 13 Peter Amstutz
# $ (cd ~/arvados/sdk/cwl && pip install -e .)
16 2 Peter Amstutz
# _do your development_
17
# $ arvados-cwl-runner --local
18 9 Eric Biagiotti
19
_Commit code, build development docker jobs image, and test submission_
20
21 2 Peter Amstutz
# $ git commit
22 13 Peter Amstutz
# $ (cd ~/arvados/sdk/cwl && pip install -e .)
23 2 Peter Amstutz
# $ ~/arvados/build/build-dev-docker-jobs-image.sh
24
# $ arvados-cwl-runner --submit
25 3 Peter Amstutz
26 9 Eric Biagiotti
_Important to note:_
27 6 Peter Amstutz
28 10 Eric Biagiotti
* You need to have ARVADOS_API_HOST and ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on _Current token_ in the user dropdown menu. 
29 7 Peter Amstutz
* "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp).
30 10 Eric Biagiotti
* If you are using arvbox, the paths here refer to your development arvados repository in the .arvbox folder
31 13 Peter Amstutz
* You can specify custom schema salad and/or cwltool source to use in the dev docker jobs image by doing the following:
32 1 Peter Amstutz
    
33 13 Peter Amstutz
     * $ SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh
34 9 Eric Biagiotti
35 13 Peter Amstutz
* Enable all the debugging output with @arvados-cwl-runner --debug@
36
* If you have a problem that can only be reproduced by running in the container environment, you can add log statements to a-c-r, cwltool and/or salad in order to get more insight as to what is happening.  Be user to lower them to 'debug' level or remove them when you are done.
37
38 12 Eric Biagiotti
h2. Testing
39 10 Eric Biagiotti
40 11 Eric Biagiotti
This section describes how to set up your testing environment and run/debug CWL conformance and Arvados integration tests. Tests can be run several different ways. You can run tests using a Python 3 virtualenv in a docker image running Python 2, vice versa, or in a pure Python 2 or 3 setup. Python 2 is the default for build-dev-docker-jobs-image.sh. Use "PYCMD=python3 arvados/build/build-dev-docker-jobs-image.sh" to get a Python 3 dev docker image. You can use the --python="path/to/python/exec/" virtualenv argument to specify your python version. You can also test both the jobs and containers APIs by changing the --api flag on run-tests.sh. 
41
42
If using arvbox, you can more easily test with test_with_arvbox.sh, which is in the arvados/sdk/cwl folder. Commands include:
43
44
--suite (conformance|integration)
45
46
--api (jobs|containers)
47
48
Example: test_with_arvbox.sh --no-reset-container --leave-running --config dev --build --pythoncmd python3 --suite integration -j3
49
50 1 Peter Amstutz
51 10 Eric Biagiotti
h3. Environment setup
52
53
# $ "arvbox start" and "arvswitch" to your dev cluster or run "arvbox start dev"
54
# If ARVADOS_API_HOST and ARVADOS_API_TOKEN are not set, find the the workbench ip address from the arvbox output. Open it in a browser and add a new account with fake credentials. Click on _Current token_ in the user dropdown menu and follow instructions. If you subsequently receive localhost match errors, use "export ARVADOS_API_HOST_INSECURE=1".
55 1 Peter Amstutz
# $ virtualenv venv
56
# $ . venv/bin/activate
57 13 Peter Amstutz
# $ (cd ~/arvados/sdk/cwl && pip install -e .)
58 10 Eric Biagiotti
# $ ~/arvados/build/build-dev-docker-jobs-image.sh.
59
60
h3. Running CWL conformance tests
61
 
62
run_tests.sh is located in the common-workflow-language repository. To improve speed, use -jx, where x is the number tests to run simultaneously. To run a single test, use -nx where x is the test number zero-indexed. 
63
64 1 Peter Amstutz
# $ pip install cwltest
65
# $ git clone https://github.com/common-workflow-language/common-workflow-language.git
66
67 10 Eric Biagiotti
_Run tests_
68 1 Peter Amstutz
69 10 Eric Biagiotti
# $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --local"
70
# $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --submit"
71
72
_Debug tests_
73
74
When a test fails, there should be output displaying _Test failed:_ followed by the test command. Copy the command. 
75
76
# $ cd common-workflow-language/v1.0
77
# $ Paste the command and replace the --quiet flag with --debug
78
79
h3. Running arvados CWL integration tests
80
81 4 Peter Amstutz
# $ cd arvados/sdk/cwl/tests
82 1 Peter Amstutz
# $ ./arvados-tests.sh