Story #12289

Migrate to Slurm API

Added by Tom Morris almost 3 years ago. Updated almost 3 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Arvados Future Sprints
Start date:
09/26/2017
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Currently nodemanager & crunch-dispatch-slurm parse the output of Slurm command line utilities to get the information that it needs. This is fragile and labor intensive since changes to the output format must be tracked by the parsing code.

We should investigate switching to using the actual Slurm API.
https://github.com/PySlurm/pyslurm

Subtasks

Task #12336: Investigate Slurm API New

History

#1 Updated by Tom Morris almost 3 years ago

  • Description updated (diff)

#2 Updated by Tom Clegg almost 3 years ago

Potential benefits include better compatibility across slurm versions (less CLI-output-parsing code) and lower overhead (less shelling out to CLI).

However, https://github.com/PySlurm/pyslurm says

You will need to instruct the setup.py script where either the Slurm install root directory or where the Slurm libraries and Slurm include files are

It sounds like this package provides an API client for a specific version of slurm, which could be just another distribution/deployment nightmare. It only makes sense if every (current+future) slurm library version is forward & backward compatible with all slurm versions. Otherwise, CLI tools are more reliable -- at least it's possible to accommodate versions with "if v3 then X else Y" when necessary.

Also available in: Atom PDF