https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422022-07-06T17:21:21ZArvadosArvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1047742022-07-06T17:21:21ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/104774/diff?detail_id=101355">diff</a>)</li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1047752022-07-06T17:21:59ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>28</i></li><li><strong>Start date</strong> set to <i>12/01/2022</i></li><li><strong>Due date</strong> set to <i>05/31/2023</i></li><li><strong>Project</strong> changed from <i>Arvados</i> to <i>Arvados Epics</i></li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1047762022-07-06T17:22:43ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Tracker</strong> changed from <i>Feature</i> to <i>Idea</i></li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1047782022-07-06T17:51:49ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-1 priority-4 priority-default" href="/issues/11161">Idea #11161</a>: [GA4GH] Support task execution API using crunch v2 containers</i> added</li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1047792022-07-06T18:27:32ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><ul>
<li>Uses a task wrapper script that's presumably built on the fly
<ul>
<li>it looks like a bash script with custom bash functions</li>
<li>Example custom bash function is nxf_s3_upload() which is a wrapper around 'aws' command line</li>
</ul>
</li>
<li>Backends provide a "FileCopyStrategy" which emit commands to the bash script which do the specific file download & upload for a task invocation</li>
<li>For example, TES: <a class="external" href="https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/ScriptFileCopyStrategy.groovy">https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/ScriptFileCopyStrategy.groovy</a></li>
<li>Or AWS: nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy</li>
</ul>
<p>The bash script gets built here: nextflow/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy</p>
<p>The driver seems to be here:</p>
<p>nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy</p>
<p>Where newSubmitRequest() actually translates "nextflow.processor.TaskRun" to "com.amazonaws.services.batch.model.SubmitJobRequest"</p> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1047802022-07-06T18:52:45ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Handles containers by building a Docker (or other container runner) command line on the fly, e.g.</p>
<p>nextflow/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy</p>
<p>Which gets injected into the do-everything bash script (in BashWrapperBuilder)</p>
<p>Seems to lean on Docker to pull the image on demand</p> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1047832022-07-06T19:13:34ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Subject</strong> changed from <i>Investigate running NextFlow on Arvados</i> to <i>Investigate running NextFlow using TES on Arvados</i></li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1094242023-01-03T16:27:31ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Start date</strong> changed from <i>12/01/2022</i> to <i>04/01/2023</i></li><li><strong>Due date</strong> changed from <i>05/31/2023</i> to <i>10/31/2023</i></li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1107992023-02-08T15:53:45ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Start date</strong> changed from <i>04/01/2023</i> to <i>01/01/2024</i></li><li><strong>Due date</strong> changed from <i>10/31/2023</i> to <i>04/30/2024</i></li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1172812023-08-15T23:12:33ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Subject</strong> changed from <i>Investigate running NextFlow using TES on Arvados</i> to <i>Investigate running NextFlow on Arvados (possibly using TES)</i></li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1208562024-01-04T15:01:58ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Start date</strong> changed from <i>01/01/2024</i> to <i>06/01/2024</i></li><li><strong>Due date</strong> changed from <i>04/30/2024</i> to <i>12/31/2024</i></li></ul> Arvados Epics - Idea #19246: Investigate running NextFlow on Arvados (possibly using TES)https://dev.arvados.org/issues/19246?journal_id=1237752024-03-01T21:33:21ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> set to <i>Future</i></li></ul>