Container dispatch » History » Revision 16
Revision 15 (Tom Clegg, 12/23/2015 03:29 PM) → Revision 16/26 (Tom Clegg, 12/23/2015 09:43 PM)
h1. Container Crunch2 dispatch {{toc}} h2. Summary A dispatcher uses available compute resources to execute queued containers. Dispatch is meant to be a small simple component rather than a pluggable framework: e.g., "slurm dispatch" can be a small standalone program, rather than a plugin for a big generic dispatch program. h2. Pseudocode * Notice there is a queued container * Decide whether the required resources are available to run the container * Lock the container (this avoids races with other dispatch processes) * Translate the container's runtime constraints and priority to instructions for the lower-level scheduler, if any * Invoke the "crunch2 run" executor * When the priority changes on a container taken by this dispatch process, update the lower-level scheduler accordingly (cancel if priority is zero) * If the lower-level scheduler indicates the container is finished or abandoned, but the Container record is locked by this dispatcher and has state=Running, fail the container h2. Examples slurm batch mode * Use "sinfo" to determine whether it is possible to run the container * Submit a batch job to the queue: "echo crunch-run --job {uuid} | sbatch -N1" * When container priority changes, use scontrol and scancel to propagate changes to slurm * Use strigger to run a cleanup script when a container exits standalone worker * Inspect /proc/meminfo, /proc/cpuinfo, "docker ps", etc. to determine local capacity * Invoke crunch-run as a child process (or perhaps a detached daemon process) * Signal crunch-run to stop if container priority changes to zero h2. Arvados API support Each dispatch process has an Arvados API token that allows it to see queued containers. * No two dispatch processes can run at the same time with the same token. One way to achieve this is to make a user record for each dispatch service. Container APIs relevant to a dispatch program: * List Queued containers (might be a subset of Queued containers) * List containers with state=Locked or state=Running associated with current token * Receive event when container is created or modified and state is Queued (it might become runnable) * Change state Queued->Locked * Change state Locked->Queued * Change state Locked->Running * Change state Running->Complete * Receive event when priority changes * Receive event when state changes to Complete * Create a unique API token to pass to crunch-run (expires when the container stops) * Create events/logs ** Decided not to run this container ** Decided to run this container (e.g., no node with those resources) ** Lock failed ** Dispatched to crunch-run ** Cleaned up crashed crunch-run (lower-level scheduler indicates the job finished, but crunch-run didn't leave the container in a final state) ** Cleaned up abandoned container (container belongs to this process, but dispatch and lower-level scheduler don't know about it) h2. Non-responsibilities Dispatch doesn't retry failed containers. If something needs to be reattempted, a new container will appear in the queue. Dispatch doesn't fail a container that it can't run. It doesn't know whether other dispatchers will be able to run it. h2. Additional notes (see also #6429 and #6518) Using websockets to listen for container events (new containers added, priority changes) will benefit from some Go SDK support.