https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422021-09-03T14:40:00ZArvadosArvados - Feature #18096: Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabledhttps://dev.arvados.org/issues/18096?journal_id=967412021-09-03T14:40:00ZWard Vandewegeward@curii.com
<ul><li><strong>Subject</strong> changed from <i>Run WGS variant calling workflow on 9tee4 with LSF and singularity enabled</i> to <i>Run WGS variant calling workflow on 9tee4 with Slurm and singularity enabled</i></li></ul> Arvados - Feature #18096: Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabledhttps://dev.arvados.org/issues/18096?journal_id=967462021-09-03T14:48:26ZWard Vandewegeward@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed parent" href="/issues/17857">Idea #17857</a>: Run WGS variant calling workflow on 9tee4 with LSF and Singularity enabled</i> added</li></ul> Arvados - Feature #18096: Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabledhttps://dev.arvados.org/issues/18096?journal_id=967472021-09-03T14:48:35ZWard Vandewegeward@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Ward Vandewege</i></li></ul> Arvados - Feature #18096: Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabledhttps://dev.arvados.org/issues/18096?journal_id=967482021-09-03T14:53:49ZWard Vandewegeward@curii.com
<ul><li><strong>Subject</strong> changed from <i>Run WGS variant calling workflow on 9tee4 with Slurm and singularity enabled</i> to <i>Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabled</i></li></ul> Arvados - Feature #18096: Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabledhttps://dev.arvados.org/issues/18096?journal_id=967502021-09-03T15:09:33ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul><p>Diag workflow worked fine with Slurm + Singularity: <a href="https://arvadosapi.com/9tee4-xvhdp-muiii2mkby0y9n9">9tee4-xvhdp-muiii2mkby0y9n9</a> and <a href="https://arvadosapi.com/9tee4-xvhdp-05oml4rackb6tb4">9tee4-xvhdp-05oml4rackb6tb4</a>.</p>
<p>Now running the big workflow:</p>
<pre>
$ arvados-cwl-runner --project-uuid <a href="https://arvadosapi.com/9tee4-j7d0g-uefv7xq2hlwyph5">9tee4-j7d0g-uefv7xq2hlwyph5</a> ./cwl/wgs-processing-wf.cwl ./yml/wgs-processing-wf-chr19.yml
INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20210827152247, arvados-python-client 2.3.0.dev20210827152247, cwltool 3.0.20210319143721
INFO Resolved './cwl/wgs-processing-wf.cwl' to 'file:///home/ward/arvados-tutorial/WGS-processing/cwl/wgs-processing-wf.cwl'
INFO Using cluster 9tee4 (https://workbench2.9tee4.arvadosapi.com/)
INFO Upload local files: "tail.html" "generatereport.py" "head.html"
INFO Uploaded to 4fce132bc4ae9b4cc844115246a6bd41+175 (<a href="https://arvadosapi.com/9tee4-4zz18-2x31x08s4433o0n">9tee4-4zz18-2x31x08s4433o0n</a>)
INFO Using collection cache size 256 MiB
INFO [container WGS processing workflow scattered over samples] submitted container_request <a href="https://arvadosapi.com/9tee4-xvhdp-5chn41x5n3793gl">9tee4-xvhdp-5chn41x5n3793gl</a>
INFO Monitor workflow progress at https://workbench2.9tee4.arvadosapi.com/processes/9tee4-xvhdp-5chn41x5n3793gl
...
INFO Final process status is success
</pre>
<p>Duh, that one reused everything of course. Trying again:</p>
<pre>
$ arvados-cwl-runner --disable-reuse --project-uuid <a href="https://arvadosapi.com/9tee4-j7d0g-uefv7xq2hlwyph5">9tee4-j7d0g-uefv7xq2hlwyph5</a> ./cwl/wgs-processing-wf.cwl ./yml/wgs-processing-wf-chr19.yml
INFO /usr/bin/arvados-cwl-runner 2.3.0.dev20210827152247, arvados-python-client 2.3.0.dev20210827152247, cwltool 3.0.20210319143721
INFO Resolved './cwl/wgs-processing-wf.cwl' to 'file:///home/ward/arvados-tutorial/WGS-processing/cwl/wgs-processing-wf.cwl'
INFO Using cluster 9tee4 (https://workbench2.9tee4.arvadosapi.com/)
INFO Upload local files: "generatereport.py" "tail.html" "head.html"
INFO Using collection 4fce132bc4ae9b4cc844115246a6bd41+175 (<a href="https://arvadosapi.com/9tee4-4zz18-2x31x08s4433o0n">9tee4-4zz18-2x31x08s4433o0n</a>)
INFO Using collection cache size 256 MiB
INFO [container WGS processing workflow scattered over samples] submitted container_request <a href="https://arvadosapi.com/9tee4-xvhdp-o0l519uaanemolg">9tee4-xvhdp-o0l519uaanemolg</a>
INFO Monitor workflow progress at https://workbench2.9tee4.arvadosapi.com/processes/9tee4-xvhdp-o0l519uaanemolg
...
INFO Final process status is success
</pre>
<p>This completed without issue. It ran much faster than the LSF + Singularity job from <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: Run WGS variant calling workflow on 9tee4 with LSF and Singularity enabled (Resolved)" href="https://dev.arvados.org/issues/17857">#17857</a>, there is clearly a performance problem with our LSF setup. Tracking that in <a class="external" href="https://dev.arvados.org/issues/17857#note-22">https://dev.arvados.org/issues/17857#note-22</a>.</p> Arvados - Feature #18096: Run WGS variant calling workflow on 9tee4 with Slurm and Singularity enabledhttps://dev.arvados.org/issues/18096?journal_id=967622021-09-03T18:46:15ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul>