Arvados

According to https://aws.amazon.com/ec2/faqs/#EC2_On-Demand_Instance_limits there are exactly 5 (or perhaps exactly 6) vCPU-based instance limits. We could determine the appropriate instance family automatically from ProviderType.

22017-instance-type-quotas @ 2989422c0e7b11c3a2d39079bb9e30779bf45bca -- developer-run-tests: #4431

• All agreed upon points are implemented / addressed.
  • The dispatcher adds a concept of "instance family" and a CapacityError interface method that lets a driver report "instance family is at capacity".
  • Each driver is responsible for reporting the instance family of a given instance type.
  • The EC2 driver reports instance family based on ProviderType, based on the AWS docs. Other drivers report a single instance family "".
  • The EC2 driver reports VcpuLimitExceeded errors as instance family specific.
• Anything not implemented (discovered or discussed during work) has a follow-up story.
  • N/A
• Code is tested and passing, both automated and manual, what manual testing was done is described
  • ✅ test case added
• Documentation has been updated.
  • N/A
• Behaves appropriately at the intended scale (describe intended scale).
  • ✅ no scaling impact anticipated
• Considered backwards and forwards compatibility issues between client and server.
  • ✅ no compatibility issues
• Follows our coding standards and GUI style guidelines.
  • ✅

Added a test case comment in 8e1aba0d8bcb6e0c5689c93868d8afba3af73d90, didn't re-run tests.

My concern is that AWS will continue to introduce new instance types that the ec2 driver doesn't know about, and then we'll have the same problem.

Is there a reason to not go for a simpler solution of using the part before the dot in the instance type names? It could drop trailing digits so you just have initial symbol, so 'm5.xlarge' takes the 'm5' and then becomes just 'm'.

Then we could describe quota groups in config file:

QuotaGroup:
  standard:
    m: {}
    c: {}
    t: {}
  x:
    x: {}
  g:
    g: {}

We can have the config default match what we know now, but retain the ability to tweak it in the future.

If an instance is listed in the config file that isn't known to QuotaGroups, it would be an error.

Here's documentation about instance type name components: https://docs.aws.amazon.com/ec2/latest/instancetypes/instance-type-names.html
Detaching type families' series from quota groups with a default group-to-families mapping sounds like a good idea.

Updated using https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-quotas.html which does include the families missing from the other doc page (hpc, trn, dl, u, vt).

Updated to track quota separately for spot and demand instances.

22017-instance-type-quotas @ 949658f3cede442bf1da4d67a2c9892a1b61945a -- developer-run-tests: #4435

It would be extra nice if AWS offered this information via API, but they don't seem to.

I think it would be more succinct to write the config map as alphaprefix→family, so the default config would be

InstanceTypeFamilies:
  a: standard
  c: standard
  d: standard
  g: g
  inf: inf
  vt: g
  ...

Alternatively, we could simply treat unrecognized alphabetic prefixes as being their own distinct instance families, instead of defaulting to "standard". It only takes one extra "create instance" attempt/error to set the "new family [that's not actually a distinct family] is also at capacity" flag, so behavior with an outdated list would be nearly indistinguishable from the desired behavior.

developer-run-tests: #4442

• rename "instance family" to "instance quota group" because "instance family" means something different in AWS
• allow configuring additional instance quota groups
• treat unrecognized alphabetic prefixes as being their own distinct instance quota groups

• for demand instances, P2, P3, P4, P5 are all in the same quota group
• for spot instances, P2, P3, P4 are all in the same quota group, and P5 is in its own quota group
• Rather than invent a config language to express this, the current implementation puts P5 in its own instance group for both spot and demand instances. This means a few avoidable failed "create" calls in the demand case, but I don't think that's a big deal.

Just curious what happens here:

1. families A and B are in the same quota group
2. the config file doesn't reflect that, so they're considered to be in separate quota groups
3. we create an instance in of type A, the quota is now full
4. we try to create an instance type B but get back "quota full" error
5. the "B" quota group is now in "at quota" state, which holds off on creating further instances until an instance in quota group "B" has been shut down

Is it now stuck forever? Under what conditions will it try again to create an instance of type B? Do we need a special case where we get a quota error but zero instances have been started?

Peter Amstutz wrote in #note-16:

Is it now stuck forever? Under what conditions will it try again to create an instance of type B?

Our "at capacity for group B" state lasts for one minute. At that point we wake up the scheduler, and (assuming it's still needed) it will try to create a new instance of type B. If the cloud provider returns another VcpuLimitExceeded error, repeat.

This might be a little confusing because although our "quota error" handling does wait for an instance shutdown before trying again, and the AWS language suggests this is a quota error, we are now treating this as a "capacity error" instead, which means we just wait 1 minute and try again.

22017-instance-type-quotas LGTM

Tom Clegg wrote in #note-17:

Peter Amstutz wrote in #note-16:

Is it now stuck forever? Under what conditions will it try again to create an instance of type B?

Our "at capacity for group B" state lasts for one minute. At that point we wake up the scheduler, and (assuming it's still needed) it will try to create a new instance of type B. If the cloud provider returns another VcpuLimitExceeded error, repeat.

This might be a little confusing because although our "quota error" handling does wait for an instance shutdown before trying again, and the AWS language suggests this is a quota error, we are now treating this as a "capacity error" instead, which means we just wait 1 minute and try again.

Thanks, I had suggested making it a capacity error in the initial writeup, I just had not looked at the code to see that was exactly what you ended up doing.