Bug #10186

[Crunch2] Support dispatching to multiple SLURM partitions

Added by Tom Morris over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/06/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0
Release:
Release relationship:
Auto

Subtasks

Task #10199: crunch-dispatch-slurm gets partition from runtime constraintsResolvedPeter Amstutz

Task #10198: Add partition to RuntimeConstraints hint in arvados-cwl-runnerResolvedPeter Amstutz

Task #10201: Review 10186-crunch2-slurm-partitionResolvedLucas Di Pentima

Associated revisions

Revision 9f35a3cf
Added by Peter Amstutz over 2 years ago

Merge branch '10186-crunch2-slurm-partition' closes #10186

History

#1 Updated by Tom Morris over 2 years ago

  • Target version set to 2016-10-12 sprint
  • Story points set to 1.0

Define new run time constraint called "Partition".

Add support to arvados dispatch slurm map to sbatch argument.

#2 Updated by Peter Amstutz over 2 years ago

  • Assigned To set to Peter Amstutz

#3 Updated by Peter Amstutz over 2 years ago

  • Status changed from New to In Progress

#4 Updated by Lucas Di Pentima over 2 years ago

The title for this ticket talks about "multiple partitions" but the code only refers to partition in singular.
Reading sbatch documentation, it seems that passing a comma separated string can be used to dispatch to multiple partitions, but at least for me it's not clear because of the constraints and variable naming.
If the objective is really to dispatch to multiple partitions, can the runtime constraint be defined as an array of strings that get translated to the appropriate comma separated string?

#5 Updated by Peter Amstutz over 2 years ago

Lucas Di Pentima wrote:

The title for this ticket talks about "multiple partitions" but the code only refers to partition in singular.
Reading squeue documentation, it seems that passing a comma separated string can be used to dispatch to multiple partitions, but at least for me it's not clear because of the constraints and variable naming.
If the objective is really to dispatch to multiple partitions, can the runtime constraint be defined as an array of strings that get translated to the appropriate comma separated string?

The intent was "select a partition among multiple partitions" instead of being restricted to either the default partition or a single explicitly configured partition in SbatchArguments. However, you're right, the more general version of this feature should allow listing multiple partitions.

I realize that this branch is also missing a documentation update. Will work on it.

#6 Updated by Peter Amstutz over 2 years ago

Now @ a5ce43cd7493e17fe1bb5f67451957b283881228

"partition" takes array of strings. Documentation is updated.

Test job: https://ci.curoverse.com/job/developer-run-tests/

#7 Updated by Lucas Di Pentima over 2 years ago

LGTM, please merge.

#8 Updated by Peter Amstutz over 2 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:9f35a3cf10e5d261d83556d5b33f242966101f02.

Also available in: Atom PDF