Arvados-cwl-runner development » History » Version 9
Eric Biagiotti, 01/28/2019 04:41 PM
1 | 1 | Peter Amstutz | h1. Arvados-cwl-runner development |
---|---|---|---|
2 | |||
3 | 9 | Eric Biagiotti | Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow. |
4 | 2 | Peter Amstutz | |
5 | 9 | Eric Biagiotti | In submit mode (--submit) it creates a container request which executes arvados-cwl-runner (a-c-r) inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow, and gets information about the container it is running in using containers().current(). |
6 | 2 | Peter Amstutz | |
7 | 9 | Eric Biagiotti | When doing development related to workflow execution, you can use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development "arvados/jobs" image with the correct code and version tag. |
8 | 1 | Peter Amstutz | |
9 | 9 | Eric Biagiotti | h2. Example development process |
10 | 2 | Peter Amstutz | |
11 | 9 | Eric Biagiotti | _Setup python virtual environment, edit code, and test locally_ |
12 | |||
13 | 1 | Peter Amstutz | # $ virtualenv venv |
14 | # $ . venv/bin/activate |
||
15 | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
||
16 | 2 | Peter Amstutz | # _do your development_ |
17 | # $ arvados-cwl-runner --local |
||
18 | 9 | Eric Biagiotti | |
19 | _Commit code, build development docker jobs image, and test submission_ |
||
20 | |||
21 | 2 | Peter Amstutz | # $ git commit |
22 | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
||
23 | # $ ~/arvados/build/build-dev-docker-jobs-image.sh |
||
24 | # $ arvados-cwl-runner --submit |
||
25 | 3 | Peter Amstutz | |
26 | 9 | Eric Biagiotti | _Important to note:_ |
27 | 6 | Peter Amstutz | |
28 | 9 | Eric Biagiotti | * You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment. This can be obtained by opening workbench and clicking on _Current token_ in the user dropdown menu. |
29 | 7 | Peter Amstutz | * "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp). |
30 | 9 | Eric Biagiotti | * If you are using arvbox, the paths above refer to your development arvados repository in the .arvbox folder |
31 | * You can specify custom schema salad, cwltool packages, and/or the python executable to use in the dev docker jobs image by doing the following: |
||
32 | |||
33 | * $ PYCMD=python_or_python3 SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh |
||
34 | 5 | Peter Amstutz | |
35 | 4 | Peter Amstutz | h2. Running cwl conformance tests |
36 | |||
37 | 8 | Peter Amstutz | # $ "arvbox start" and "arvswitch" to your dev cluster |
38 | 4 | Peter Amstutz | # $ virtualenv venv |
39 | # $ . venv/bin/activate |
||
40 | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
||
41 | # $ pip install cwltest |
||
42 | # $ git clone https://github.com/common-workflow-language/common-workflow-language.git |
||
43 | # $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse" |
||
44 | |||
45 | h2. Running arvados integration tests |
||
46 | |||
47 | # $ virtualenv venv |
||
48 | # $ . venv/bin/activate |
||
49 | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
||
50 | # $ cd arvados/sdk/cwl/tests |
||
51 | # $ ./arvados-tests.sh |