Arvados-cwl-runner development » History » Version 6
Peter Amstutz, 01/11/2019 09:35 PM
1 | 1 | Peter Amstutz | h1. Arvados-cwl-runner development |
---|---|---|---|
2 | |||
3 | 2 | Peter Amstutz | Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow |
4 | |||
5 | In submit mode (--submit) it creates a container request which executes arvados-cwl-runner inside a container. The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance. The a-c-r instance inside the container actively manages the workflow (--local), and gets information about the container it is running in using containers().current(). |
||
6 | |||
7 | When doing development related to workflow execution, you can also use "--local" to run a locally managed workflow using your development code. However, you still need to test that the workflow can be submitted and run in the container. This requires building a development arvados/jobs image with the correct code and version tag. |
||
8 | |||
9 | Here's a summary of the development process: |
||
10 | |||
11 | 1 | Peter Amstutz | # $ virtualenv venv |
12 | # $ . venv/bin/activate |
||
13 | 2 | Peter Amstutz | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
14 | 1 | Peter Amstutz | # _do your development_ |
15 | 2 | Peter Amstutz | # _test locally_ |
16 | # $ arvados-cwl-runner --local |
||
17 | 1 | Peter Amstutz | # $ git commit |
18 | 2 | Peter Amstutz | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
19 | # $ ~/arvados/build/build-dev-docker-jobs-image.sh |
||
20 | # _test submit to cluster_ |
||
21 | # $ arvados-cwl-runner --submit |
||
22 | |||
23 | Important to note: |
||
24 | |||
25 | * You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment |
||
26 | * "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp). |
||
27 | 3 | Peter Amstutz | * the a-c-r code which is run by the submitter and which is copied in the docker image are expected to match, and it uses the version number of the python package / docker image to enforce that. |
28 | 4 | Peter Amstutz | |
29 | 6 | Peter Amstutz | h2. Building a jobs image with custom cwltool or schema-salad |
30 | 1 | Peter Amstutz | |
31 | 6 | Peter Amstutz | You can build a dev jobs image with custom schema salad and/or cwltool packages this way: |
32 | |||
33 | $ SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh |
||
34 | 5 | Peter Amstutz | |
35 | 4 | Peter Amstutz | h2. Running cwl conformance tests |
36 | |||
37 | # $ virtualenv venv |
||
38 | # $ . venv/bin/activate |
||
39 | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
||
40 | # $ pip install cwltest |
||
41 | # $ git clone https://github.com/common-workflow-language/common-workflow-language.git |
||
42 | # $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse" |
||
43 | |||
44 | h2. Running arvados integration tests |
||
45 | |||
46 | # $ virtualenv venv |
||
47 | # $ . venv/bin/activate |
||
48 | # $ (cd ~/arvados/sdk/cwl && python setup.py develop) |
||
49 | # $ cd arvados/sdk/cwl/tests |
||
50 | # $ ./arvados-tests.sh |