Arvados

20667-maxsuper-atquota @ c624929279e70d58017eab08ed286dda88bcd215 -- developer-run-tests: #3720

This takes AtQuota() into account when applying the max-supervisor limit, and adds an at_quota metric.

Also, when hitting quota, the scheduler reduces its self-imposed concurrency level to the current level, so it only gets raised 10% at a time after that point. Essentially, once it sees its first quota error, its self-imposed limit will approximately track the cloud quota, even if the cloud quota raises (e.g., the quota is based on #CPUs so the effective instance limit is variable).

However, this still doesn't address the following fairly obvious/likely scenario:

1. Set MaxInstances=1000, MaxSupervisors=0.4
2. Queue has 400 supervisors when a-d-c starts up
3. Scheduler starts 250 supervisors successfully
4. Cloud reports quota error
5. Scheduler reduces max supervisors target to 100, but it's too late
6. Deadlock

I think we could address this by having a configurable initial value for the self-imposed concurrency limit (right now it's unlimited). A low initial value (say 8) would give a-d-c a "soft start" effect where it waits for the first few supervisors to start some child containers before pushing up the max_concurrent_containers limit and admitting more supervisors.

InitialQuotaEstimate or InitialConcurrencyLimit or ...?

20667-maxsuper-atquota @ 5372ee7878d6880081dd4b5481e1820fc7cd1975 -- developer-run-tests: #3722

With InitialQuotaEstimate config.

Tom Clegg wrote in #note-6:

20667-maxsuper-atquota @ 5372ee7878d6880081dd4b5481e1820fc7cd1975 -- developer-run-tests: #3722

With InitialQuotaEstimate config.

This LGTM.

Sorry it took forever for me to get to.