Project

General

Profile

Container dispatch » History » Version 21

Peter Amstutz, 05/02/2016 07:11 PM

1 16 Tom Clegg
h1. Container dispatch
2 2 Peter Amstutz
3 15 Tom Clegg
{{toc}}
4 9 Peter Amstutz
5 15 Tom Clegg
h2. Summary
6 1 Peter Amstutz
7 15 Tom Clegg
A dispatcher uses available compute resources to execute queued containers.
8 1 Peter Amstutz
9 15 Tom Clegg
Dispatch is meant to be a small simple component rather than a pluggable framework: e.g., "slurm dispatch" can be a small standalone program, rather than a plugin for a big generic dispatch program.
10 12 Peter Amstutz
11 15 Tom Clegg
h2. Pseudocode
12 1 Peter Amstutz
13 15 Tom Clegg
* Notice there is a queued container
14
* Decide whether the required resources are available to run the container
15
* Lock the container (this avoids races with other dispatch processes)
16
* Translate the container's runtime constraints and priority to instructions for the lower-level scheduler, if any
17
* Invoke the "crunch2 run" executor
18
* When the priority changes on a container taken by this dispatch process, update the lower-level scheduler accordingly (cancel if priority is zero)
19
* If the lower-level scheduler indicates the container is finished or abandoned, but the Container record is locked by this dispatcher and has state=Running, fail the container
20 1 Peter Amstutz
21 15 Tom Clegg
h2. Examples
22 1 Peter Amstutz
23 15 Tom Clegg
slurm batch mode
24
* Use "sinfo" to determine whether it is possible to run the container
25
* Submit a batch job to the queue: "echo crunch-run --job {uuid} | sbatch -N1"
26
* When container priority changes, use scontrol and scancel to propagate changes to slurm
27
* Use strigger to run a cleanup script when a container exits
28 2 Peter Amstutz
29 15 Tom Clegg
standalone worker
30
* Inspect /proc/meminfo, /proc/cpuinfo, "docker ps", etc. to determine local capacity
31
* Invoke crunch-run as a child process (or perhaps a detached daemon process)
32
* Signal crunch-run to stop if container priority changes to zero
33 2 Peter Amstutz
34 15 Tom Clegg
h2. Arvados API support
35 2 Peter Amstutz
36 15 Tom Clegg
Each dispatch process has an Arvados API token that allows it to see queued containers.
37
* No two dispatch processes can run at the same time with the same token. One way to achieve this is to make a user record for each dispatch service.
38 2 Peter Amstutz
39 15 Tom Clegg
Container APIs relevant to a dispatch program:
40
* List Queued containers (might be a subset of Queued containers)
41
* List containers with state=Locked or state=Running associated with current token
42 18 Tom Clegg
** arvados.v1.containers.current (equivalent to @filters=[["dispatch_auth_uuid","=",current_client_auth.uuid]]@)
43 15 Tom Clegg
* Receive event when container is created or modified and state is Queued (it might become runnable)
44
* Change state Queued->Locked
45
* Change state Locked->Queued
46
* Change state Locked->Running
47
* Change state Running->Complete
48
* Receive event when priority changes
49 1 Peter Amstutz
* Receive event when state changes to Complete
50 18 Tom Clegg
* Retrieve an API token to pass into the container and its arv-mount process (via crunch-run)
51
** Token is automatically created/assigned when container state changes to Locked
52
** Token is automatically expired/destroyed when container state changes away from Running
53 19 Tom Clegg
** arvados.v1.containers.container_auth(uuid=container.uuid) → returns an api_client_authorization record
54 15 Tom Clegg
* Create events/logs
55
** Decided not to run this container
56
** Decided to run this container (e.g., no node with those resources)
57
** Lock failed
58
** Dispatched to crunch-run
59
** Cleaned up crashed crunch-run (lower-level scheduler indicates the job finished, but crunch-run didn't leave the container in a final state)
60
** Cleaned up abandoned container (container belongs to this process, but dispatch and lower-level scheduler don't know about it)
61 6 Peter Amstutz
62 15 Tom Clegg
h2. Non-responsibilities
63 6 Peter Amstutz
64 15 Tom Clegg
Dispatch doesn't retry failed containers. If something needs to be reattempted, a new container will appear in the queue.
65 7 Peter Amstutz
66 15 Tom Clegg
Dispatch doesn't fail a container that it can't run. It doesn't know whether other dispatchers will be able to run it.
67 8 Peter Amstutz
68 15 Tom Clegg
h2. Additional notes
69 8 Peter Amstutz
70 17 Tom Clegg
(see also #6429 and #6518 and #8028)
71 8 Peter Amstutz
72 15 Tom Clegg
Using websockets to listen for container events (new containers added, priority changes) will benefit from some Go SDK support.
73 20 Peter Amstutz
74
h2. Cloud Container Services
75
76
Cloud providers now offer container execution services.  However, rather than being just an API to run containers (similar to Crunch) these take the form of preconfigured clusters set up with a container orchestration system.
77
78
AWS offers Elastic Container Service.  It appears that the leader runs on AWS infrastructure (?) and you spin up worker VMs which run the ECS Agent: https://github.com/aws/amazon-ecs-agent
79
80 21 Peter Amstutz
Google Container Engine provides a preconfigured Kubernetes cluster.  https://cloud.google.com/container-engine/docs/clusters/operations
81 20 Peter Amstutz
82 21 Peter Amstutz
Azure provides a preconfigured Mesos or Docker Swarm cluster.  https://azure.microsoft.com/en-us/services/container-service/?WT.mc_id=azurebg_email_Trans_1083_Tier2_Release_MOSP