[arvados-dispatch-cloud] kill crunch-run procs for containers that are deleted or have state=Cancelled when dispatcher starts up
Currently, a container that has state==Cancelled when arvados-dispatch-cloud starts up will never be added to the container queue, even if its UUID appears on an instance's probe result. Also, a container that has been deleted from the database will never have an entry added/updated in the dispatcher's container queue.
The scheduler's sync() func is responsible for killing unneeded crunch-run processes, but it only looks at the container queue, so these crunch-run processes are allowed to run forever.
Proposed solution:In (*scheduler.Scheduler)sync(), kill anything returned by sch.pool.Running() that isn't returned by sch.queue.Entries(). This should be safe from "kill crunch-run before seeing its UUID in the queue" races:
- at least one "get entire queue from controller/database" has succeeded before the first call to sync()
- UUIDs are added to Running() only during (*Scheduler)runQueue(), which does not run concurrently with (*Scheduler)sync().
In (*container.Queue)poll(), if a container's UUID is in the local queue but is not returned by the API calls that request that specific UUID, delete it from the local queue. The "get missing containers" loop will need to be more careful to avoid accidentally deleting containers when the API server chooses to return less than a full page of results.
14977-kill-if-not-in-queue @ 2a748e79c3a72454d70e40f39fcad9dabf4943cc
This also fixes a different startup bug: "fix stale locks" was not waiting for the pool to load its initial instance list, so it would always return immediately, and the scheduler would create too many new instances at startup ("container is locked but no instances are available to run it").
Successfully tested on c97qk.
If a container is already cancelled when the dispatcher starts up, it never gets added/updated in the dispatcher's queue. Therefore, the scheduler never finds out it has state=Cancelled. So the fix is to kill any container that is not in the queue -- whether because Cancelled, because deleted, or whatever.