Arvados » Arvados Epics

• Uses a task wrapper script that's presumably built on the fly
  • it looks like a bash script with custom bash functions
  • Example custom bash function is nxf_s3_upload() which is a wrapper around 'aws' command line
• Backends provide a "FileCopyStrategy" which emit commands to the bash script which do the specific file download & upload for a task invocation
• For example, TES: https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/ScriptFileCopyStrategy.groovy
• Or AWS: nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy

The bash script gets built here: nextflow/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy

The driver seems to be here:

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy

Where newSubmitRequest() actually translates "nextflow.processor.TaskRun" to "com.amazonaws.services.batch.model.SubmitJobRequest"

Handles containers by building a Docker (or other container runner) command line on the fly, e.g.

nextflow/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy

Which gets injected into the do-everything bash script (in BashWrapperBuilder)

Seems to lean on Docker to pull the image on demand