Story #9464

[Crunch2] Test Curoverse Crunch 2 installation

Added by Brett Smith almost 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
-
Target version:
Start date:
08/03/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

On 9tee4, after Crunch 2 is deployed and running there:

  • Simple "hello world" container
  • Simple single-container workflow
  • Simple CWL workflow
  • More complex CWL workflow (but remember there are only two compute nodes)

Subtasks

Task #9467: Run testsResolvedRadhika Chippada

History

#1 Updated by Brett Smith almost 5 years ago

  • Assigned To set to Peter Amstutz

#2 Updated by Peter Amstutz almost 5 years ago

  • Story points set to 0.5

#3 Updated by Radhika Chippada almost 5 years ago

  • Target version changed from 2016-07-06 sprint to 2016-07-20 sprint

#4 Updated by Brett Smith almost 5 years ago

  • Assigned To deleted (Peter Amstutz)

#5 Updated by Brett Smith almost 5 years ago

  • Description updated (diff)

#6 Updated by Brett Smith almost 5 years ago

  • Description updated (diff)

#7 Updated by Radhika Chippada almost 5 years ago

  • Assigned To set to Radhika Chippada
  • Story points changed from 0.5 to 1.0

#8 Updated by Brett Smith almost 5 years ago

This is unblocked. As discussed before, you will need to create container requests that specify ram and vcpus as positive integers. ram is a number of bytes; vpcus is a number of cores. The compute nodes on the test cluster have 32GB RAM and 20 cores.

#9 Updated by Radhika Chippada almost 5 years ago

  • Target version changed from 2016-07-20 sprint to 2016-08-03 sprint

#10 Updated by Radhika Chippada almost 5 years ago

Success so far:

  • Created a container_request 9tee4-xvhdp-kfd3hb2toubtjmp with priority 1 and state Committed (and other required attributes). This was successfully completed.
  • arvados-cwl-runner --api containers bwa-mem.cwl bwa-mem-input.yml succeeded as well
  • arvados-cwl-runner --api containers --no-wait bwa-mem.cwl bwa-mem-input.yml also succeeded
  • arvados-cwl-runner --api containers --local bwa-mem.cwl bwa-mem-input.yml also succeeded

Failing as of now:

  • arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml is failing due to missing keep ref (possibly due to being out of sync with latest updates). Filed #9659
    (venv)1!radhika@shell.9tee4:~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner bwa-mem.cwl bwa-mem-input.yml
    /home/radhika/venv/bin/arvados-cwl-runner 1.0.20160717133709, arvados-python-client 0.1.20160627200331, cwltool 1.0.20160714182449
    2016-07-21 19:43:30 arvados.arv-run[13106] INFO: Upload local files: "bwa-mem.cwl" 
    2016-07-21 19:43:30 arvados.arv-run[13106] INFO: Using collection 9tee4-4zz18-1om5a5o9jxspigw
    2016-07-21 19:43:30 arvados.cwl-runner[13106] ERROR: Caught unhandled exception, marking pipeline as failed.  Error was: 'UploadFile' object has no attribute 'keepref'
    Workflow error, try again with --debug for more information:
      Workflow failed.
    
  • arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml failed as well.
    (venv)1!radhika@shell.9tee4:~/arvados/doc/user/cwl/bwa-mem$ arvados-cwl-runner --local bwa-mem.cwl bwa-mem-input.yml
    /home/radhika/venv/bin/arvados-cwl-runner 1.0.20160717133709, arvados-python-client 0.1.20160627200331, cwltool 1.0.20160714182449
    2016-07-21 19:50:26 arvados.cwl-runner[13977] INFO: Pipeline instance 9tee4-d1hrv-lpkqxtrpk8r3wu8
    2016-07-21 19:51:54 arvados.cwl-runner[13977] ERROR: Got error <HttpError 422 when requesting https://9tee4.arvadosapi.com/arvados/v1/jobs?alt=json&find_or_create=true&filters=%5B%5B%22repository%22%2C+%22%3D%22%2C+%22arvados%22%5D%2C+%5B%22script%22%2C+%22%3D%22%2C+%22crunchrunner%22%5D%2C+%5B%22script_version%22%2C+%22in+git%22%2C+%229e5b98e8f5f4727856b53447191f9c06e3da2ba6%22%5D%2C+%5B%22docker_image_locator%22%2C+%22in+docker%22%2C+%220cb15ebcfec378e2b746e74c7d6e037a%2B840%22%5D%5D returned "#<Commit::GitError: echo e63f502086fa0cfd1570b3bb02416abba9186307 2>&1 |git --git-dir /var/lib/gitolite/repositories/9tee4-s0uqq-36ruocozt6ilsx7.git pack-objects -q --revs --stdout 2>&1 |git --git-dir /var/lib/arvados/internal.git unpack-objects -q 2>&1: pid 17634 exit 128: error: insufficient permission for adding an object to repository database /var/lib/arvados/internal.git/objects
    
    fatal: failed to write objec
    

#11 Updated by Radhika Chippada almost 5 years ago

  • Status changed from New to In Progress

#12 Updated by Radhika Chippada almost 5 years ago

  • Simple "hello world" container

Passed. https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-kfd3hb2toubtjmp

  • Simple single-container workflow

Passed. I took the bwa-mem cwa workflow and configured it as a simple container request. https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-f1zgya7m1ouxd4b

  • Simple CWL workflow

Passed. Used "arvados-cwl-runner --local --api containers bwa-mem.cwl bwa-mem-input.yml" from the doc/user/cwl/bwa-mem directory. https://workbench.9tee4.arvadosapi.com/container_requests/9tee4-xvhdp-0o4fxhwrqmnn7xz

  • More complex CWL workflow (but remember there are only two compute nodes)

Passed with Crunch1. Initially, I used Sally's unc workflow but later on switched to Jiayong's varscan cwl workflow (main-goCallVars.cwl) using the command "arvados-cwl-runner --local --wait --debug main-goCallVars.cwl test-input-yml/main-goCallVars-job.yml". https://workbench.9tee4.arvadosapi.com/pipeline_instances/9tee4-d1hrv-wgvlm9an4a6caw6 . I also ran using "--api containers", in which case I see only two container_requests get created and pass (freebayes and varscan) and the freebayes CR failing. Created #9703

#13 Updated by Radhika Chippada almost 5 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF