Dispatching containers to cloud VMs » History » Revision 7
Revision 6 (Tom Clegg, 09/05/2018 09:08 PM) → Revision 7/82 (Tom Clegg, 09/19/2018 02:50 PM)
h1. Dispatching containers to cloud VMs (Draft) h2. Component name / purpose crunch-dispatch-cloud runs Arvados user containers on generic public cloud infrastructure by automatically creating (TBD) crunch-dispatch-cloud, or arvados-dispatch-cloud, or arvados-dispatch (c-d-slurm and destroying VMs of various sizes according to demand, preparing the VMs' c-d-local could become arvados-dispatch modules, selected at runtime environments, and running containers on them. via config, rather than shipping as separate packages/programs). h2. Overview The dispatcher waits for containers to appear in the queue, and runs them on appropriately sized cloud VMs. When there are no idle cloud VMs with the desired size, the dispatcher brings up more VMs using the cloud provider's API. The dispatcher also shuts down idle VMs that exceed the configured idle timer -- and sooner if the provider refuses to create new VMs. h2. Interaction with other components API server (backed by PostgreSQL) supplies the container queue: which containers the system should be trying to execute (or cancel) at any given time. The cloud provider's API supplies a list of VMs that exist (or are being created) at a given time and their network addresses, accepts orders to create new VMs, updates instance tags, and (optionally, depending on the driver) obtains the VMs' SSH server public keys. The SSH server on each cloud VM allows the dispatcher to authenticate with a private key and execute shell commands as root. h2. Configuration Arvados configuration (currently a file in /etc) supplies cloud provider credentials, allowed node types, spending limits/policies, etc. <pre><code class="yaml"> CloudVMs: BootTimeout: 20m Driver: Amazon DriverParameters: Region: us-east-1 APITimeout: 20s EC2Key: abcdef EC2Secret: abcdefghijklmnopqrstuvwxyz StorageKey: abcdef StorageSecret: abcdefghijklmnopqrstuvwxyz ImageID: ami-0123456789abcdef0 SubnetID: subnet-01234567 SecurityGroups: sg-01234567 </code></pre> h2. Scheduling policy The container priority field determines the order in which resources are allocated. * If container C1 has priority P1, * ...and C2 has higher priority P2, * ...and there is no pending/booting/idle VM suitable for running C2, * ...then C1 will not be started. However, containers that run on different VM types don't necessarily start in priority order. * If container C1 has priority P1, * ...and C2 has higher priority P2, * ...and there is no idle VM suitable for running C2, * ...and there is a pending/booting VM that will be suitable for running C2 when it comes up, * ...and there is an idle VM suitable for running C1, * ...then C1 will start before C2. h2. Synchronizing state When first starting up, dispatcher inspects API server’s container queue and the cloud provider’s list of dispatcher-tagged cloud nodes, and restores internal state accordingly. Often, at startup there will be some containers with state=Locked. To avoid breaking priority order, the dispatcher won't schedule any new containers until all such locked containers are matched up with crunch-run processes on existing VMs (typically preparing a docker image) or all of the existing VMs have been probed successfully (meaning the locked containers aren't running anywhere and need to be rescheduled). When a user cancels a container request with state=Locked or Running, the container priority changes to 0. On its next poll, the dispatcher notices this and kills any corresponding crunch-run processes (or, if there is no such process, just unlocks the container). When a crunch-run process ends without finalizing its container's state, the dispatcher notices this and sets state to Cancelled. h2. Operator view Management status endpoint provides: * list of cloud VMs, each with ** provider's instance ID ** hourly price (from configuration file) ** instance type (from configuration file) ** instance type (from provider's menu) ** UUID of the current / most recent container attempted (if known) ** time last container finished (or boot time, if nothing run yet) * list of queued/running containers, each with ** UUID ** state (queued/locked/running/complete/cancelled) ** desired instance type ** time appeared in queue ** time started (if started) Metrics endpoint tracks: * (each VM) time elapsed between VM creation and first successful SSH connection * (each VM) time elapsed between first successful SSH connection and ready to run a container * total hourly price of all existing VMs * total VCPUs and memory allocated to containers * number of containers running * number of containers allocated to VMs but not started yet (because VMs are pending/booting) * number of containers not allocated to VMs (because provider quota is reached) h2. SSH keys Each worker node has a public key in /root/.ssh/authorized_keys. Dispatcher has the corresponding private key. (Future) Dispatcher generates its own keys and installs its public key on new VMs using cloud provider bootstrapping/metadata features. h3. Probes Sometimes (on the happy path) the dispatcher knows the state of each worker, whether it's idle, and which container it's running. In general, it's necessary to probe the worker node itself. Probe: * Check whether the SSH connection is alive; reopen if needed. * Run the configured "ready?" command (e.g., "grep /encrypted-tmp /etc/mtab"); if this fails, conclude the node is still booting. * Run "crunch-run --list" to get a list of crunch-run supervisors (pid + container UUID) Dispatcher, after a successful probe, should tag the cloud node record with the dispatcher's ID and probe timestamp. (In case the tagging API fails, remember the probe time in memory too.) h3. Detecting dead/lame nodes If a node has been up for N seconds without a successful probe, despite at least M attempts, shut it down. (M handles the case where the dispatcher restarts during a time when the "update tags" operation isn't effective, e.g., provider is rate-limiting API calls.) h3. Multiple dispatchers Not supported in initial version.