Bug #7479

[Node Manager] Aggressively throttles node creation requests when the cloud region is full

Added by Brett Smith almost 5 years ago. Updated 8 months ago.

Assigned To:
Node Manager
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


Functional requirements:

  • Node Manager recognizes when attempts to create nodes fail because the cloud provider indicates that the region is full and cannot create more nodes.
    • This logic is cloud-specific. If it's difficult to write the logic for all our supported clouds, treat this story as the S3 story, and create stories to add recognition logic for other cloud providers.
  • When Node Manager receives that response, it aggressively throttles back its attempts to create more nodes. It ensures that it is only sending API requests to create a single node, and the API requests to create that node have exponential backoff. (The backoff should already be built in.) Do not send API requests for additional nodes beyond that first until the first is finally successfully created.


#1 Updated by Tom Morris over 3 years ago

  • Assigned To set to Tom Morris
  • Target version set to Arvados Future Sprints

#2 Updated by Tom Morris about 2 years ago

  • Assigned To deleted (Tom Morris)
  • Target version changed from Arvados Future Sprints to To Be Groomed

#3 Updated by Peter Amstutz 8 months ago

  • Target version deleted (To Be Groomed)
  • Status changed from New to Closed

Also available in: Atom PDF