Bug #16795
closed
Added by Tom Clegg over 4 years ago.
Updated over 4 years ago.
Release relationship:
Auto
Description
run-tests-remainder: #3934 
/consoleText
dispatcher_test.go:212:
c.Check(resp.Body.String(), check.Matches, `(?ms).*boot_outcomes{outcome="aborted"} 0.*`)
...
... "arvados_dispatchcloud_boot_outcomes{outcome=\"aborted\"} 15\n" +
I think this started in
#16739 when we fixed
(*worker.Pool)Create() to return false when a rate-limit hold is in effect.
- In the simulation test, the stub driver returns a rate-limit error when it gets two Create() calls within 1ms (MinTimeBetweenInstancesCalls).
- When this happens, (*Pool)Create() calls wp.instanceSet.throttleCreate.CheckRateLimitError()
- The next call to Create() (if it happens in the same 1ms window) returns false because the rate-limit hold is still in effect.
- When Create() returns false, the scheduler invokes its quota-avoidance strategy: avoid starting any lower-priority containers, and shut down any idle/booting nodes that aren't needed for higher-priority containers.
- When rescheduling a container due to a simulated error or instance failure, there's likely to be a booting instance whose type only matches lower-priority containers, requested back when the previous attempt on the higher-priority container hadn't failed yet.
- When this happens, the booting instances get shut down, and end up with boot outcome "aborted".
Solution: Stop invoking the quota-avoidance strategy when pool.Create() returns false but pool.AtQuota() is false. (When cloud providers/drivers have been returning errors other than quota/rate-limit, Create() returns true and tries creating instances asynchronously, so this change only affects the rate-limiting case.)
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by Anonymous 
Updated by
Merge branch '16795-boot-outcome-aborted'
fixes #16795
Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@tomclegg.ca>