Project

General

Profile

Arvados-cwl-runner development » History » Version 5

Peter Amstutz, 08/10/2018 03:36 PM

1 1 Peter Amstutz
h1. Arvados-cwl-runner development
2
3 2 Peter Amstutz
Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow
4
5
In submit mode (--submit) it creates a container request which executes arvados-cwl-runner inside a container.  The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance.  The a-c-r instance inside the container actively manages the workflow (--local), and gets information about the container it is running in using containers().current().
6
7
When doing development related to workflow execution, you can also use "--local" to run a locally managed workflow using your development code.  However, you still need to test that the workflow can be submitted and run in the container.  This requires building a development arvados/jobs image with the correct code and version tag.
8
9
Here's a summary of the development process:
10
11 1 Peter Amstutz
# $ virtualenv venv
12
# $ . venv/bin/activate
13 2 Peter Amstutz
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
14 1 Peter Amstutz
# _do your development_
15 2 Peter Amstutz
# _test locally_
16
# $ arvados-cwl-runner --local
17 1 Peter Amstutz
# $ git commit
18 2 Peter Amstutz
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
19
# $ ~/arvados/build/build-dev-docker-jobs-image.sh
20
# _test submit to cluster_
21
# $ arvados-cwl-runner --submit
22
23
Important to note:
24
25
* You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment
26
* "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp).
27 3 Peter Amstutz
* the a-c-r code which is run by the submitter and which is copied in the docker image are expected to match, and it uses the version number of the python package / docker image to enforce that.
28 4 Peter Amstutz
29 5 Peter Amstutz
h2. Testing changes to Arvados Python SDK, cwltool or schema-salad
30
31
If changes 
32
33 4 Peter Amstutz
h2. Running cwl conformance tests
34
35
# $ virtualenv venv
36
# $ . venv/bin/activate
37
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
38
# $ pip install cwltest
39
# $ git clone https://github.com/common-workflow-language/common-workflow-language.git
40
# $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse"
41
42
h2. Running arvados integration tests
43
44
# $ virtualenv venv
45
# $ . venv/bin/activate
46
# $ (cd ~/arvados/sdk/cwl && python setup.py develop)
47
# $ cd arvados/sdk/cwl/tests
48
# $ ./arvados-tests.sh