Idea #19246
open
Investigate running NextFlow on Arvados (possibly using TES)
Description
NextFlow supports a variety of backends, including AWS Batch, Azure Batch and TES , all of which exist at a similar level and with similar capabilities to the Arvados Crunch container request API.
https://www.nextflow.io/docs/latest/executor.html
Investigate potential for a Crunch backend for NextFlow and/or a TES compatibility layer for Arvados.
Related issues
Updated by Peter Amstutz over 2 years ago
- Release set to 28
- Start date set to 12/01/2022
- Due date set to 05/31/2023
- Project changed from Arvados to Arvados Epics
Updated by Peter Amstutz over 2 years ago
- Related to Idea #11161: [GA4GH] Support task execution API using crunch v2 containers added
Updated by Peter Amstutz over 2 years ago
- Uses a task wrapper script that's presumably built on the fly
- it looks like a bash script with custom bash functions
- Example custom bash function is nxf_s3_upload() which is a wrapper around 'aws' command line
- Backends provide a "FileCopyStrategy" which emit commands to the bash script which do the specific file download & upload for a task invocation
- For example, TES: https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/ScriptFileCopyStrategy.groovy
- Or AWS: nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy
The bash script gets built here: nextflow/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy
The driver seems to be here:
nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy
Where newSubmitRequest() actually translates "nextflow.processor.TaskRun" to "com.amazonaws.services.batch.model.SubmitJobRequest"
Updated by Peter Amstutz over 2 years ago
Handles containers by building a Docker (or other container runner) command line on the fly, e.g.
nextflow/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy
Which gets injected into the do-everything bash script (in BashWrapperBuilder)
Seems to lean on Docker to pull the image on demand
Updated by Peter Amstutz over 2 years ago
- Subject changed from Investigate running NextFlow on Arvados to Investigate running NextFlow using TES on Arvados
Updated by Peter Amstutz almost 2 years ago
- Start date changed from 12/01/2022 to 04/01/2023
- Due date changed from 05/31/2023 to 10/31/2023
Updated by Peter Amstutz almost 2 years ago
- Start date changed from 04/01/2023 to 01/01/2024
- Due date changed from 10/31/2023 to 04/30/2024
Updated by Peter Amstutz over 1 year ago
- Subject changed from Investigate running NextFlow using TES on Arvados to Investigate running NextFlow on Arvados (possibly using TES)
Updated by Peter Amstutz 11 months ago
- Start date changed from 01/01/2024 to 06/01/2024
- Due date changed from 04/30/2024 to 12/31/2024