Arvados

I think this started in #16739 when we fixed (*worker.Pool)Create() to return false when a rate-limit hold is in effect.

• In the simulation test, the stub driver returns a rate-limit error when it gets two Create() calls within 1ms (MinTimeBetweenInstancesCalls).
• When this happens, (*Pool)Create() calls wp.instanceSet.throttleCreate.CheckRateLimitError()
• The next call to Create() (if it happens in the same 1ms window) returns false because the rate-limit hold is still in effect.
• When Create() returns false, the scheduler invokes its quota-avoidance strategy: avoid starting any lower-priority containers, and shut down any idle/booting nodes that aren't needed for higher-priority containers.
• When rescheduling a container due to a simulated error or instance failure, there's likely to be a booting instance whose type only matches lower-priority containers, requested back when the previous attempt on the higher-priority container hadn't failed yet.
• When this happens, the booting instances get shut down, and end up with boot outcome "aborted".

Solution: Stop invoking the quota-avoidance strategy when pool.Create() returns false but pool.AtQuota() is false. (When cloud providers/drivers have been returning errors other than quota/rate-limit, Create() returns true and tries creating instances asynchronously, so this change only affects the rate-limiting case.)

16795-boot-outcome-aborted @ b30659d514ce281209fa7b99863413832fa8d44b -- developer-run-tests: #2063

(also includes a fix for a race in the stub driver that caused occasional test failures)

16795-boot-outcome-aborted @ ac4599592d265dc5a922ec8f468d46cfe7de52e2 -- developer-run-tests: #2064

(pinned pycurl to avoid the latest py3-only versions)

Tom Clegg wrote:

16795-boot-outcome-aborted @ ac4599592d265dc5a922ec8f468d46cfe7de52e2 -- developer-run-tests: #2064

(pinned pycurl to avoid the latest py3-only versions)

LGTM, thanks for the detailed explanation in note 1. The pycurl pin should be removed as part of #15888.