Bug #7479

[Node Manager] Aggressively throttles node creation requests when the cloud region is full

Added by Brett Smith about 3 years ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Node Manager
Target version:
Start date:
10/07/2015
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Functional requirements:

  • Node Manager recognizes when attempts to create nodes fail because the cloud provider indicates that the region is full and cannot create more nodes.
    • This logic is cloud-specific. If it's difficult to write the logic for all our supported clouds, treat this story as the S3 story, and create stories to add recognition logic for other cloud providers.
  • When Node Manager receives that response, it aggressively throttles back its attempts to create more nodes. It ensures that it is only sending API requests to create a single node, and the API requests to create that node have exponential backoff. (The backoff should already be built in.) Do not send API requests for additional nodes beyond that first until the first is finally successfully created.

History

#1 Updated by Tom Morris about 2 years ago

  • Assigned To set to Tom Morris
  • Target version set to Arvados Future Sprints

#2 Updated by Tom Morris 4 months ago

  • Assigned To deleted (Tom Morris)
  • Target version changed from Arvados Future Sprints to To Be Groomed

Also available in: Atom PDF