Project

General

Profile

Actions
Copy link

Feature #18096

closed

Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabled

Added by Ward Vandewege over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Ward Vandewege
Category:
-
Target version:
2021-09-15 sprint
Story points:
-

Related issues

Related to Arvados - Idea #17857: Run WGS variant calling workflow on 9tee4 with LSF and Singularity enabledResolvedWard Vandewege08/27/2021Actions
Actions
Copy link
 #1

Updated by Ward Vandewege over 2 years ago

  • Subject changed from Run WGS variant calling workflow on 9tee4 with LSF and singularity enabled to Run WGS variant calling workflow on 9tee4 with Slurm and singularity enabled
Actions
Copy link
 #2

Updated by Ward Vandewege over 2 years ago

  • Related to Idea #17857: Run WGS variant calling workflow on 9tee4 with LSF and Singularity enabled added
Actions
Copy link
 #3

Updated by Ward Vandewege over 2 years ago

  • Assigned To set to Ward Vandewege
Actions
Copy link
 #4

Updated by Ward Vandewege over 2 years ago

  • Subject changed from Run WGS variant calling workflow on 9tee4 with Slurm and singularity enabled to Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabled
Actions
Copy link
 #5

Updated by Ward Vandewege over 2 years ago

  • Status changed from New to In Progress

Diag workflow worked fine with Slurm + Singularity: 9tee4-xvhdp-muiii2mkby0y9n9 and 9tee4-xvhdp-05oml4rackb6tb4.

Now running the big workflow:

$ arvados-cwl-runner --project-uuid 9tee4-j7d0g-uefv7xq2hlwyph5 ./cwl/wgs-processing-wf.cwl ./yml/wgs-processing-wf-chr19.yml
INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20210827152247, arvados-python-client 2.3.0.dev20210827152247, cwltool 3.0.20210319143721
INFO Resolved './cwl/wgs-processing-wf.cwl' to 'file:///home/ward/arvados-tutorial/WGS-processing/cwl/wgs-processing-wf.cwl'
INFO Using cluster 9tee4 (https://workbench2.9tee4.arvadosapi.com/)
INFO Upload local files: "tail.html" "generatereport.py" "head.html" 
INFO Uploaded to 4fce132bc4ae9b4cc844115246a6bd41+175 (9tee4-4zz18-2x31x08s4433o0n)
INFO Using collection cache size 256 MiB
INFO [container WGS processing workflow scattered over samples] submitted container_request 9tee4-xvhdp-5chn41x5n3793gl
INFO Monitor workflow progress at https://workbench2.9tee4.arvadosapi.com/processes/9tee4-xvhdp-5chn41x5n3793gl
...
INFO Final process status is success

Duh, that one reused everything of course. Trying again:

$ arvados-cwl-runner --disable-reuse --project-uuid 9tee4-j7d0g-uefv7xq2hlwyph5 ./cwl/wgs-processing-wf.cwl ./yml/wgs-processing-wf-chr19.yml
INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20210827152247, arvados-python-client 2.3.0.dev20210827152247, cwltool 3.0.20210319143721
INFO Resolved './cwl/wgs-processing-wf.cwl' to 'file:///home/ward/arvados-tutorial/WGS-processing/cwl/wgs-processing-wf.cwl'
INFO Using cluster 9tee4 (https://workbench2.9tee4.arvadosapi.com/)
INFO Upload local files: "generatereport.py" "tail.html" "head.html" 
INFO Using collection 4fce132bc4ae9b4cc844115246a6bd41+175 (9tee4-4zz18-2x31x08s4433o0n)
INFO Using collection cache size 256 MiB
INFO [container WGS processing workflow scattered over samples] submitted container_request 9tee4-xvhdp-o0l519uaanemolg
INFO Monitor workflow progress at https://workbench2.9tee4.arvadosapi.com/processes/9tee4-xvhdp-o0l519uaanemolg
...

INFO Final process status is success

This completed without issue. It ran much faster than the LSF + Singularity job from #17857, there is clearly a performance problem with our LSF setup. Tracking that in https://dev.arvados.org/issues/17857#note-22.

Actions
Copy link
 #6

Updated by Ward Vandewege over 2 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF