Feature #8028

Updated by Tom Clegg almost 6 years ago

This is a stub to support development and testing of other Crunch2 components. It isn't meant to be production-ready for this story, although in future work it may become part of a production setup and serve as a template for writing new dispatchers.

the "standalone worker" example described at [[Container dispatch]], without the "determine local capacity" part: it just takes all containers that appear in the queue, and starts them immediately as child processes. part.

The implementation should emphasize simplicity: we don't want to load this up with features. Anything non-trivial probably needs to be done in a different layer anyway (e.g., API server or crunch-run).

h2. Implementation

"crunch-dispatch-local", a Go program in source:services/crunch-dispatch-local/
* The main goroutine polls
Simple dispatcher which monitors the queue every 10 (configurable with "-poll-interval 5s") seconds using containers table, claims a time.NewTicker(). When a new container appears, pass container, and runs it to a "run" goroutine.
locally. The "run" goroutine starts "crunch-run" (configurable with "-crunch-run-command /usr/bin/crunch-run") and waits until [a] the crunch-run command exits, or [b] the container's priority changes goal of this story is to zero. If sketch out the latter happens, send TERM to dispatcher architecture without the crunch-run command, complications of SLURM, and keep waiting until it exits.
* Emit the relevant logs mentioned at [[Container dispatch]].

It's not very efficient for each "run" goroutine
to poll the API server about its own container record, but it's not necessary to optimize this right now. (Ideally, when we are ready to optimize this, we'll go straight to websocket updates rather than batching our polling have a dispatcher that is useful for all containers Arvados crunch development and testing that are running now, etc.) does not depend on SLURM.