Arvados

Functional requirements:

• Node Manager recognizes when attempts to create nodes fail because the cloud provider indicates that the region is full and cannot create more nodes.
  • This logic is cloud-specific. If it's difficult to write the logic for all our supported clouds, treat this story as the S3 story, and create stories to add recognition logic for other cloud providers.
• When Node Manager receives that response, it aggressively throttles back its attempts to create more nodes. It ensures that it is only sending API requests to create a single node, and the API requests to create that node have exponential backoff. (The backoff should already be built in.) Do not send API requests for additional nodes beyond that first until the first is finally successfully created.