Arvados-cwl-runner development » History » Version 13
Peter Amstutz, 10/13/2021 08:25 PM
1 | 1 | Peter Amstutz | h1. Arvados-cwl-runner development |
---|---|---|---|
2 | |||
3 | 9 | Eric Biagiotti | Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow. |
4 | 2 | Peter Amstutz | |
5 | 9 | Eric Biagiotti | In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow, and gets information about the container it is running in using containers().current(). |
6 | 2 | Peter Amstutz | |
7 | 9 | Eric Biagiotti | When doing development related to workflow execution, you can use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development "arvados/jobs" image with the correct code and version tag. |
8 | 1 | Peter Amstutz | |
9 | 9 | Eric Biagiotti | h2. Example development process |
10 | 2 | Peter Amstutz | |
11 | 9 | Eric Biagiotti | _Setup python virtual environment, edit code, and test locally_ |
12 | |||
13 | 1 | Peter Amstutz | # $ virtualenv venv |
14 | # $ . venv/bin/activate |
||
15 | 13 | Peter Amstutz | # $ (cd ~/arvados/sdk/cwl && pip install -e .) |
16 | 2 | Peter Amstutz | # _do your development_ |
17 | # $ arvados-cwl-runner --local |
||
18 | 9 | Eric Biagiotti | |
19 | _Commit code, build development docker jobs image, and test submission_ |
||
20 | |||
21 | 2 | Peter Amstutz | # $ git commit |
22 | 13 | Peter Amstutz | # $ (cd ~/arvados/sdk/cwl && pip install -e .) |
23 | 2 | Peter Amstutz | # $ ~/arvados/build/build-dev-docker-jobs-image.sh |
24 | # $ arvados-cwl-runner --submit |
||
25 | 3 | Peter Amstutz | |
26 | 9 | Eric Biagiotti | _Important to note:_ |
27 | 6 | Peter Amstutz | |
28 | 10 | Eric Biagiotti | * You need to have ARVADOS_API_HOST and ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on _Current token_ in the user dropdown menu. |
29 | 7 | Peter Amstutz | * "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp). |
30 | 10 | Eric Biagiotti | * If you are using arvbox, the paths here refer to your development arvados repository in the .arvbox folder |
31 | 13 | Peter Amstutz | * You can specify custom schema salad and/or cwltool source to use in the dev docker jobs image by doing the following: |
32 | 1 | Peter Amstutz | |
33 | 13 | Peter Amstutz | * $ SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh |
34 | 9 | Eric Biagiotti | |
35 | 13 | Peter Amstutz | * Enable all the debugging output with @arvados-cwl-runner --debug@ |
36 | * If you have a problem that can only be reproduced by running in the container environment, you can add log statements to a-c-r, cwltool and/or salad in order to get more insight as to what is happening. Be user to lower them to 'debug' level or remove them when you are done. |
||
37 | |||
38 | 12 | Eric Biagiotti | h2. Testing |
39 | 10 | Eric Biagiotti | |
40 | 11 | Eric Biagiotti | This section describes how to set up your testing environment and run/debug CWL conformance and Arvados integration tests. Tests can be run several different ways. You can run tests using a Python 3 virtualenv in a docker image running Python 2, vice versa, or in a pure Python 2 or 3 setup. Python 2 is the default for build-dev-docker-jobs-image.sh. Use "PYCMD=python3 arvados/build/build-dev-docker-jobs-image.sh" to get a Python 3 dev docker image. You can use the --python="path/to/python/exec/" virtualenv argument to specify your python version. You can also test both the jobs and containers APIs by changing the --api flag on run-tests.sh. |
41 | |||
42 | If using arvbox, you can more easily test with test_with_arvbox.sh, which is in the arvados/sdk/cwl folder. Commands include: |
||
43 | |||
44 | --suite (conformance|integration) |
||
45 | |||
46 | --api (jobs|containers) |
||
47 | |||
48 | Example: test_with_arvbox.sh --no-reset-container --leave-running --config dev --build --pythoncmd python3 --suite integration -j3 |
||
49 | |||
50 | 1 | Peter Amstutz | |
51 | 10 | Eric Biagiotti | h3. Environment setup |
52 | |||
53 | # $ "arvbox start" and "arvswitch" to your dev cluster or run "arvbox start dev" |
||
54 | # If ARVADOS_API_HOST and ARVADOS_API_TOKEN are not set, find the the workbench ip address from the arvbox output. Open it in a browser and add a new account with fake credentials. Click on _Current token_ in the user dropdown menu and follow instructions. If you subsequently receive localhost match errors, use "export ARVADOS_API_HOST_INSECURE=1". |
||
55 | 1 | Peter Amstutz | # $ virtualenv venv |
56 | # $ . venv/bin/activate |
||
57 | 13 | Peter Amstutz | # $ (cd ~/arvados/sdk/cwl && pip install -e .) |
58 | 10 | Eric Biagiotti | # $ ~/arvados/build/build-dev-docker-jobs-image.sh. |
59 | |||
60 | h3. Running CWL conformance tests |
||
61 | |||
62 | run_tests.sh is located in the common-workflow-language repository. To improve speed, use -jx, where x is the number tests to run simultaneously. To run a single test, use -nx where x is the test number zero-indexed. |
||
63 | |||
64 | 1 | Peter Amstutz | # $ pip install cwltest |
65 | # $ git clone https://github.com/common-workflow-language/common-workflow-language.git |
||
66 | |||
67 | 10 | Eric Biagiotti | _Run tests_ |
68 | 1 | Peter Amstutz | |
69 | 10 | Eric Biagiotti | # $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --local" |
70 | # $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse --submit" |
||
71 | |||
72 | _Debug tests_ |
||
73 | |||
74 | When a test fails, there should be output displaying _Test failed:_ followed by the test command. Copy the command. |
||
75 | |||
76 | # $ cd common-workflow-language/v1.0 |
||
77 | # $ Paste the command and replace the --quiet flag with --debug |
||
78 | |||
79 | h3. Running arvados CWL integration tests |
||
80 | |||
81 | 4 | Peter Amstutz | # $ cd arvados/sdk/cwl/tests |
82 | 1 | Peter Amstutz | # $ ./arvados-tests.sh |