Feature #18096

Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabled

Added by Ward Vandewege about 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Related issues

Related to Arvados - Story #17857: Run WGS variant calling workflow on 9tee4 with LSF and Singularity enabledResolved08/27/2021

History

#1 Updated by Ward Vandewege about 2 months ago

  • Subject changed from Run WGS variant calling workflow on 9tee4 with LSF and singularity enabled to Run WGS variant calling workflow on 9tee4 with Slurm and singularity enabled

#2 Updated by Ward Vandewege about 2 months ago

  • Related to Story #17857: Run WGS variant calling workflow on 9tee4 with LSF and Singularity enabled added

#3 Updated by Ward Vandewege about 2 months ago

  • Assigned To set to Ward Vandewege

#4 Updated by Ward Vandewege about 2 months ago

  • Subject changed from Run WGS variant calling workflow on 9tee4 with Slurm and singularity enabled to Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabled

#5 Updated by Ward Vandewege about 2 months ago

  • Status changed from New to In Progress

Diag workflow worked fine with Slurm + Singularity: 9tee4-xvhdp-muiii2mkby0y9n9 and 9tee4-xvhdp-05oml4rackb6tb4.

Now running the big workflow:

$ arvados-cwl-runner --project-uuid 9tee4-j7d0g-uefv7xq2hlwyph5 ./cwl/wgs-processing-wf.cwl ./yml/wgs-processing-wf-chr19.yml
INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20210827152247, arvados-python-client 2.3.0.dev20210827152247, cwltool 3.0.20210319143721
INFO Resolved './cwl/wgs-processing-wf.cwl' to 'file:///home/ward/arvados-tutorial/WGS-processing/cwl/wgs-processing-wf.cwl'
INFO Using cluster 9tee4 (https://workbench2.9tee4.arvadosapi.com/)
INFO Upload local files: "tail.html" "generatereport.py" "head.html" 
INFO Uploaded to 4fce132bc4ae9b4cc844115246a6bd41+175 (9tee4-4zz18-2x31x08s4433o0n)
INFO Using collection cache size 256 MiB
INFO [container WGS processing workflow scattered over samples] submitted container_request 9tee4-xvhdp-5chn41x5n3793gl
INFO Monitor workflow progress at https://workbench2.9tee4.arvadosapi.com/processes/9tee4-xvhdp-5chn41x5n3793gl
...
INFO Final process status is success

Duh, that one reused everything of course. Trying again:

$ arvados-cwl-runner --disable-reuse --project-uuid 9tee4-j7d0g-uefv7xq2hlwyph5 ./cwl/wgs-processing-wf.cwl ./yml/wgs-processing-wf-chr19.yml
INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20210827152247, arvados-python-client 2.3.0.dev20210827152247, cwltool 3.0.20210319143721
INFO Resolved './cwl/wgs-processing-wf.cwl' to 'file:///home/ward/arvados-tutorial/WGS-processing/cwl/wgs-processing-wf.cwl'
INFO Using cluster 9tee4 (https://workbench2.9tee4.arvadosapi.com/)
INFO Upload local files: "generatereport.py" "tail.html" "head.html" 
INFO Using collection 4fce132bc4ae9b4cc844115246a6bd41+175 (9tee4-4zz18-2x31x08s4433o0n)
INFO Using collection cache size 256 MiB
INFO [container WGS processing workflow scattered over samples] submitted container_request 9tee4-xvhdp-o0l519uaanemolg
INFO Monitor workflow progress at https://workbench2.9tee4.arvadosapi.com/processes/9tee4-xvhdp-o0l519uaanemolg
...

INFO Final process status is success

This completed without issue. It ran much faster than the LSF + Singularity job from #17857, there is clearly a performance problem with our LSF setup. Tracking that in https://dev.arvados.org/issues/17857#note-22.

#6 Updated by Ward Vandewege about 2 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF