Project

General

Profile

Actions

Bug #10186

closed

[Crunch2] Support dispatching to multiple SLURM partitions

Added by Tom Morris over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
1.0
Release:
Release relationship:
Auto

Subtasks 3 (0 open3 closed)

Task #10199: crunch-dispatch-slurm gets partition from runtime constraintsResolvedPeter Amstutz10/06/2016Actions
Task #10198: Add partition to RuntimeConstraints hint in arvados-cwl-runnerResolvedPeter Amstutz10/06/2016Actions
Task #10201: Review 10186-crunch2-slurm-partitionResolvedLucas Di Pentima10/07/2016Actions
Actions #1

Updated by Tom Morris over 7 years ago

  • Target version set to 2016-10-12 sprint
  • Story points set to 1.0

Define new run time constraint called "Partition".

Add support to arvados dispatch slurm map to sbatch argument.

Actions #2

Updated by Peter Amstutz over 7 years ago

  • Assigned To set to Peter Amstutz
Actions #3

Updated by Peter Amstutz over 7 years ago

  • Status changed from New to In Progress
Actions #4

Updated by Lucas Di Pentima over 7 years ago

The title for this ticket talks about "multiple partitions" but the code only refers to partition in singular.
Reading sbatch documentation, it seems that passing a comma separated string can be used to dispatch to multiple partitions, but at least for me it's not clear because of the constraints and variable naming.
If the objective is really to dispatch to multiple partitions, can the runtime constraint be defined as an array of strings that get translated to the appropriate comma separated string?

Actions #5

Updated by Peter Amstutz over 7 years ago

Lucas Di Pentima wrote:

The title for this ticket talks about "multiple partitions" but the code only refers to partition in singular.
Reading squeue documentation, it seems that passing a comma separated string can be used to dispatch to multiple partitions, but at least for me it's not clear because of the constraints and variable naming.
If the objective is really to dispatch to multiple partitions, can the runtime constraint be defined as an array of strings that get translated to the appropriate comma separated string?

The intent was "select a partition among multiple partitions" instead of being restricted to either the default partition or a single explicitly configured partition in SbatchArguments. However, you're right, the more general version of this feature should allow listing multiple partitions.

I realize that this branch is also missing a documentation update. Will work on it.

Actions #6

Updated by Peter Amstutz over 7 years ago

Now @ a5ce43cd7493e17fe1bb5f67451957b283881228

"partition" takes array of strings. Documentation is updated.

Test job: https://ci.curoverse.com/job/developer-run-tests/

Actions #7

Updated by Lucas Di Pentima over 7 years ago

LGTM, please merge.

Actions #8

Updated by Peter Amstutz over 7 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:9f35a3cf10e5d261d83556d5b33f242966101f02.

Actions

Also available in: Atom PDF