Project

General

Profile

Actions

Story #19246

open

Investigate running NextFlow using TES on Arvados

Added by Peter Amstutz about 1 month ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
12/01/2022
Due date:
05/31/2023 (Due in about 10 months)
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

NextFlow supports a variety of backends, including AWS Batch, Azure Batch and TES , all of which exist at a similar level and with similar capabilities to the Arvados Crunch container request API.

https://www.nextflow.io/docs/latest/executor.html

Investigate potential for a Crunch backend for NextFlow and/or a TES compatibility layer for Arvados.


Related issues

Related to Arvados - Story #11161: [GA4GH] Support task execution API using crunch v2 containersNew

Actions
Actions #1

Updated by Peter Amstutz about 1 month ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz about 1 month ago

  • Release set to 28
  • Start date set to 12/01/2022
  • Due date set to 05/31/2023
  • Project changed from Arvados to Arvados Epics
Actions #3

Updated by Peter Amstutz about 1 month ago

  • Tracker changed from Feature to Story
Actions #4

Updated by Peter Amstutz about 1 month ago

  • Related to Story #11161: [GA4GH] Support task execution API using crunch v2 containers added
Actions #5

Updated by Peter Amstutz about 1 month ago

The bash script gets built here: nextflow/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy

The driver seems to be here:

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy

Where newSubmitRequest() actually translates "nextflow.processor.TaskRun" to "com.amazonaws.services.batch.model.SubmitJobRequest"

Actions #6

Updated by Peter Amstutz about 1 month ago

Handles containers by building a Docker (or other container runner) command line on the fly, e.g.

nextflow/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy

Which gets injected into the do-everything bash script (in BashWrapperBuilder)

Seems to lean on Docker to pull the image on demand

Actions #7

Updated by Peter Amstutz about 1 month ago

  • Subject changed from Investigate running NextFlow on Arvados to Investigate running NextFlow using TES on Arvados
Actions

Also available in: Atom PDF