Story #14807

Updated by Tom Clegg over 2 years ago

Issues encountered & fixed/worked around during dev deploy:
* -Include instance address (host or IP) in logs and management API responses-
* -Ensure @crunch-run --list@ works even if /var/lock is a symlink-
* -Log full instance ID, not (Instance)String(), which might be an abbreviated name-
* -Fix management API endpoints to allow specifying instance IDs that have slashes-
* -Pass SSH public key to Azure so it doesn't crash (Azure refuses to create a node without adding an admin account)-
* -Fix host part of SSH target address being dropped-
* -Allow driver to specify a login username-
* -Send ARVADOS_API_* values on stdin instead of environment vars (typical SSH server is configured to refuse these env vars)-
* -If ProviderType is not given in an instance type in the cluster config, default to the type name (not the empty string)-
* -Pass a random string to Azure driver as "node-token" (or fix Azure driver so it doesn't expect that)-

Further improvements necessary to run in production:
* -Send detached crunch-run stdout+stderr to systemd journal so sysadmin can make subsequent arrangements if needed-
* -Metrics: total cost of nodes in idle or booting state-
* -Metrics: total cost of nodes with admin-hold flag set-
* -Log when an instance goes down unexpectedly (i.e., state != Shutdown when deleted from list)-
* -Log when a container is added to or dropped from the queue-
* -Obey logging format in cluster config file (as of #14325, HTTP request logs were JSON, operational logs were text)-
* Send SIGKILL if container process still running after several SIGTERM attempts / N seconds after first SIGTERM
* Shutdown node if container process still running after several SIGKILL attempts
* Provide a "mark node as broken" callback mechanism for crunch-run (drain node, unless it's already marked "hold" -- see #14807#note-20)
* Configurable rate limit for Create and Destroy calls to cloud API (background: reaching API call rate limits can cause penalties; also, when multiple instance types are created concurrently, the cloud might create the lower-priority types but then reach quota before creating the higher-priority types; see #14360#note-36)
* Metrics: number of containers, split by state and instance type
* Load API host & token from cluster config file instead of env vars
* Ensure crunch-run exits instead of hanging if ARVADOS_API_HOST/TOKEN is empty or broken
* Kill containers (or at least log a warning) if a worker is kept busy by a container whose UUID does not exist according to the API server's queue (e.g., container deleted from database)
* "Kill instance now" management API
* (Azure) error out if AddedScratch>0 because that isn't implemented yet

Improvements that are desired, but not necessary to run in production (noted here for clarity until they move to their own tickets):
* -crunch-run --detach: retrieve stdout/stderr during probe, and show it in dispatcher logs- (logs go to journal instead)
* -crunch-run --detach: cleanup old stdout/stderr- (logs go to journal instead)
* -Move "cat .../node-token" host key verification mechanism out of Azure driver (instead, have the dispatcher do this itself if the driver returns cloud.ErrNotImplemented)-
* Metrics that indicate cloud failure (time we’ve spent trying but failing to create a new instance)
* Test suite that uses a real cloud provider
* Test activity/resource usage metrics
* Multiple cloud drivers
* Generic driver test suite
* Performance metrics for dispatching (e.g., time between seeing a container in the queue and starting its crunch-run process on a worker)
* Configurable spending limits
* Update runtime_status field when cancelling containers after crunch-run crashes or the cloud VM dies without finalizing the container (already done for the “no suitable instance type” case)
* If present, use VM image ID given in runtime_constraints instead of image ID from cluster config file
* (API) Allow admin users to specify image ID in runtime_constraints
* Metrics: count unexpected shutdowns, split by instance type
* Atomically install correct version of crunch-run (perhaps /proc/self/exe) to worker VM as part of boot probe
* Run crunch-run as a non-root user

Improvements that might never be implemented at all (noted here for clarity):
* Periodic status reports in logs. This kind of logging should normally (always?) be handled by an external monitoring system that connects to the existing metrics endpoint.
* Cancel containers that take longer than a configurable time limit to schedule (e.g., no nodes ever come up). Unsure whether this is useful: maybe containers should just stay queued until the problem is fixed.

[[Dispatching containers to cloud VMs]]