Arvados

22314-dispatch-rsc @ 5b0e5a375ce34ea60ffb5543e2c6bae03fc4b126

Needs some tests. Also it accounts for GPUs but doesn't actually allocate them explicitly, which may be necessary (something needs to know which GPU devices are busy and allocate a not-busy one).

I think it would be helpful to mention the problems or use-cases this is meant to address.

Tom Clegg wrote in #note-6:

I think it would be helpful to mention the problems or use-cases this is meant to address.

The current implementation has a fixed concurrency of 8 processes, with no logic to either scale it to the resources of the host nor account for the resource request of the containers. This makes it unsuitable for real work.

I was discussing with Sasha and Zoe how to have a single node install that is more suitable for doing real work. They were talking about setting up SLURM and crunch-dispatch-slurm, and I thought that sounded like a lot of complexity for a single node install, and that the only reason not to use crunch-dispatch-local is the lack of resource management as described above. So I spent a couple hours in the evening putting this branch together to do basic resource management.

Long term, making arvados-dispatch-cloud support allocating multiple containers to a node and using the "loopback" driver would make crunch-dispatch-local redundant, but as it stands today that's likely to take several weeks/months to fully implement because it needs to solve the general problem, whereas implementing the feature in crunch-dispatch-local only took a couple hours.

All that said, Sasha suggested that they might want to use SLURM anyway because they have some additional requirements for the Arvados appliance that were not mentioned in the previous conversation, but that was after I'd already started the branch.

Peter Amstutz wrote in #note-7:

a single node install that is more suitable for doing real work

That is helpful context, thanks.

Presumably we'll still do #14922 in due course.

(I take it the branch is not ready for review)

This work was done in #21926 because the use case (using GPUs on local hardware) required resource management to prevent multiple tasks from using the GPU at once.