Project

General

Profile

Actions

Feature #14021

open

[crunch-dispatch-slurm] option to set job priority directly instead of using nice values

Added by Tom Clegg about 6 years ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

Background: crunch-dispatch-slurm uses slurm's "nice" feature to make its slurm jobs' priority order match the priority order of their corresponding arvados containers. This is better than setting slurm job priorities directly in that it doesn't require c-d-s to have slurm administrator privileges. However, on older versions of slurm, its effectiveness is limited by slurm's maximum nice value of 10000.

crunch-dispatch-slurm could offer a configuration flag that causes it to set slurm job priority directly ("scontrol update jobid=X priority=Y") instead of restricting itself to non-privileged "nice" operations.

This configuration would be useful on systems running slurm 15 where it is acceptable to give slurm administrator privileges to crunch-dispatch-slurm.

Possible implementation strategies:
  1. Assume Arvados jobs are the only ones running. Choose a high priority like 2^32-3 (slurm's usual "TOP" priority) for the highest-priority Arvados job, and count down from there using the existing PrioritySpread logic.
  2. Wait for each submitted job to appear in the slurm queue, and note its automatically assigned priority. When updating priorities, start at the highest auto-assigned priority of all jobs still shown in squeue, and count down from there using the existing PrioritySpread logic.
    • Caveat: on a cloud installation, jobs frequently drop into "admin hold" state (priority=0), so it might be some time before c-d-s gets a chance to see a given job's automatically assigned priority. The resulting behavior might be confusing.
Actions #2

Updated by Tom Clegg about 6 years ago

  • Target version set to To Be Groomed
Actions #3

Updated by Tom Clegg about 6 years ago

  • Subject changed from [crunch-dispatch-slurm] Config option to set job priority directly instead of using nice values to [crunch-dispatch-slurm] option to set job priority directly instead of using nice values
Actions #4

Updated by Peter Amstutz over 3 years ago

  • Target version deleted (To Be Groomed)
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Release set to 60
Actions #6

Updated by Peter Amstutz 8 months ago

  • Target version set to Future
Actions

Also available in: Atom PDF