Project

General

Profile

Arvados-cwl-runner development » History » Revision 7

Revision 6 (Peter Amstutz, 01/11/2019 09:35 PM) → Revision 7/14 (Peter Amstutz, 01/11/2019 09:36 PM)

h1. Arvados-cwl-runner development 

 Arvados-cwl-runner has two modes: submitting a workflow, and actively managing a workflow 

 In submit mode (--submit) it creates a container request which executes arvados-cwl-runner inside a container.    The container request uses the docker image "arvados/jobs" and the docker tag is the version of submitting a-c-r instance.    The a-c-r instance inside the container actively manages the workflow (--local), and gets information about the container it is running in using containers().current(). 

 When doing development related to workflow execution, you can also use "--local" to run a locally managed workflow using your development code.    However, you still need to test that the workflow can be submitted and run in the container.    This requires building a development arvados/jobs image with the correct code and version tag. 

 Here's a summary of the development process: 

 # $ virtualenv venv 
 # $ . venv/bin/activate 
 # $ (cd ~/arvados/sdk/cwl && python setup.py develop) 
 # _do your development_ 
 # _test locally_ 
 # $ arvados-cwl-runner --local 
 # $ git commit 
 # $ (cd ~/arvados/sdk/cwl && python setup.py develop) 
 # $ ~/arvados/build/build-dev-docker-jobs-image.sh 
 # _test submit to cluster_ 
 # $ arvados-cwl-runner --submit 

 Important to note: 

 * You need to have ARVADOS_API_HOST an ARVADOS_API_TOKEN in your environment 
 * "setup.py develop" is convenient for editing files in place, but whenever you commit to git remember to re-run setup.py to record the new version number (which is derived from the git commit timestamp). 
 * the a-c-r code which is run by the submitter and which is copied in the docker image are expected to match, and it uses the version number of the python package / docker image to enforce that. 

 h2. Building a jobs image with custom cwltool or schema-salad 

 When you You can build a "dev" dev jobs image, it generates custom arvados-python-client and arvados-cwl-runner packages from the working directory and injects them into the docker image build. 

 You can also specify with custom schema salad and/or cwltool packages this way: 

 $ SALAD=/path/to/schema-salad CWLTOOL=/path/to/cwltool ~/arvados/build/build-dev-docker-jobs-image.sh 

 h2. Running cwl conformance tests 

 # $ virtualenv venv 
 # $ . venv/bin/activate 
 # $ (cd ~/arvados/sdk/cwl && python setup.py develop) 
 # $ pip install cwltest 
 # $ git clone https://github.com/common-workflow-language/common-workflow-language.git 
 # $ ./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse" 

 h2. Running arvados integration tests 

 # $ virtualenv venv 
 # $ . venv/bin/activate 
 # $ (cd ~/arvados/sdk/cwl && python setup.py develop) 
 # $ cd arvados/sdk/cwl/tests 
 # $ ./arvados-tests.sh