Arvados

This is unblocked. As discussed before, you will need to create container requests that specify ram and vcpus as positive integers. ram is a number of bytes; vpcus is a number of cores. The compute nodes on the test cluster have 32GB RAM and 20 cores.

Success so far:

• Created a container_request 9tee4-xvhdp-kfd3hb2toubtjmp with priority 1 and state Committed (and other required attributes). This was successfully completed.

• arvados-cwl-runner --api containers bwa-mem.cwl bwa-mem-input.yml succeeded as well

• arvados-cwl-runner --api containers --no-wait bwa-mem.cwl bwa-mem-input.yml also succeeded

• arvados-cwl-runner --api containers --local bwa-mem.cwl bwa-mem-input.yml also succeeded

Failing as of now:

• arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml is failing due to missing keep ref (possibly due to being out of sync with latest updates). Filed #9659
  

  (venv)1!radhika@shell.9tee4:~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml
  /home/radhika/venv/bin/arvados-cwl-runner 1.0.20160717133709, arvados-python-client 0.1.20160627200331, cwltool 1.0.20160714182449
  2016-07-21 19:43:30 arvados.arv-run[13106] INFO: Upload local files: "bwa-mem.cwl" 
  2016-07-21 19:43:30 arvados.arv-run[13106] INFO: Using collection 9tee4-4zz18-1om5a5o9jxspigw
  2016-07-21 19:43:30 arvados.cwl-runner[13106] ERROR: Caught unhandled exception, marking pipeline as failed.  Error was: 'UploadFile' object has no attribute 'keepref'
  Workflow error, try again with --debug for more information:
    Workflow failed.
  

• arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml failed as well.
  

  (venv)1!radhika@shell.9tee4:~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml
  /home/radhika/venv/bin/arvados-cwl-runner 1.0.20160717133709, arvados-python-client 0.1.20160627200331, cwltool 1.0.20160714182449
  2016-07-21 19:50:26 arvados.cwl-runner[13977] INFO: Pipeline instance 9tee4-d1hrv-lpkqxtrpk8r3wu8
  2016-07-21 19:51:54 arvados.cwl-runner[13977] ERROR: Got error <HttpError 422 when requesting https://9tee4.arvadosapi.com/arvados/v1/jobs?alt=json&find_or_create=true&filters=%5B%5B%22repository%22%2C+%22%3D%22%2C+%22arvados%22%5D%2C+%5B%22script%22%2C+%22%3D%22%2C+%22crunchrunner%22%5D%2C+%5B%22script_version%22%2C+%22in+git%22%2C+%229e5b98e8f5f4727856b53447191f9c06e3da2ba6%22%5D%2C+%5B%22docker_image_locator%22%2C+%22in+docker%22%2C+%220cb15ebcfec378e2b746e74c7d6e037a%2B840%22%5D%5D returned "#<Commit::GitError: echo e63f502086fa0cfd1570b3bb02416abba9186307 2>&1 |git --git-dir /var/lib/gitolite/repositories/9tee4-s0uqq-36ruocozt6ilsx7.git pack-objects -q --revs --stdout 2>&1 |git --git-dir /var/lib/arvados/internal.git unpack-objects -q 2>&1: pid 17634 exit 128: error: insufficient permission for adding an object to repository database /var/lib/arvados/internal.git/objects
  
  fatal: failed to write objec
  

• Simple "hello world" container

Passed. https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-kfd3hb2toubtjmp

• Simple single-container workflow

Passed. I took the bwa-mem cwa workflow and configured it as a simple container request. https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-f1zgya7m1ouxd4b

• Simple CWL workflow

Passed. Used "arvados-cwl-runner --local --api containers bwa-mem.cwl bwa-mem-input.yml" from the doc/user/cwl/bwa-mem directory. https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-0o4fxhwrqmnn7xz

• More complex CWL workflow (but remember there are only two compute nodes)

Passed with Crunch1. Initially, I used Sally's unc workflow but later on switched to Jiayong's varscan cwl workflow (main-goCallVars.cwl) using the command "arvados-cwl-runner --local --wait --debug main-goCallVars.cwl test-input-yml/main-goCallVars-job.yml". https://workbench.9tee4.arvadosapi.com/pipeline_instances/9tee4-d1hrv-wgvlm9an4a6caw6 . I also ran using "--api containers", in which case I see only two container_requests get created and pass (freebayes and varscan) and the freebayes CR failing. Created #9703