Idea #9581
closed[Crunch2] SLURM dispatcher supports arbitrary arguments to sbatch
Description
Use case¶
Administrators want additional control over how Arvados containers are dispatched to SLURM. Some options they might commonly want to specify are --account
, --clusters
, or --partition
. crunch-dispatch-slurm should provide a simple mechanism for administrators to set arbitrary additional arguments to sbatch.
Implementation¶
Add a command line flag, "-config path/to/config.json" (see source:services/keep-balance/main.go for an example). crunch-dispatch-slurm reads configuration entries from this JSON file. If no -config
switch is given, it tries to read configuration from /etc/arvados/crunch-dispatch-slurm/config.json
. If that file does not exist, crunch-dispatch-slurm continues running with default configuration. If there's any other error reading the file (permissions problem, invalid JSON), or if the user specified a config file that doesn't exist, that's a fatal error, and the program should abort with a useful error message.
Convert existing command line flags to config file entries.
type Config struct { SbatchArguments []string PollPeriod time.Duration CrunchRunCommand string }
The configured SbatchArguments should be inserted into the command passed to exec.Command()
, after the generic options sbatch --share
and before the job-specific options --job-name
, --mem-per-cpu
, and --cpus-per-task
. This is because the administrator's configuration should take precedence over default behavior switches, but not over job-specific switches which have more details about how the container should be run.
Tests¶
Just unit tests for sbatchFunc.- Handles nil SbatchArguments correctly
- Handles non-nil SbatchArguments correctly