Bug #12318

Honor Retry-After headers on libcloud exceptions

Added by Lucas Di Pentima almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Node Manager
Target version:
Start date:
09/25/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

It seems to be a bug on libcloud that could make nodemanager behave erratically on certain error situations with cloud providers.
As for libcloud 2.2.1, it seems that neither BaseHTTPError nor RateLimitReachedError get assigned the Retry-After header value, as it was removed from the exception_from_message() call on libcloud/libcloud/common/base.py.

Furthermore, the Retry-After header should accept a date in the future as detailed on https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html (relevant part copied below):

The Retry-After response-header field can be used with a 503 (Service Unavailable) response to indicate how long the service is expected to be unavailable to the requesting client. This field MAY also be used with any 3xx (Redirection) response to indicate the minimum time the user-agent is asked wait before issuing the redirected request. The value of this field can be either an HTTP-date or an integer number of seconds (in decimal) after the time of the response.

       Retry-After  = "Retry-After" ":" ( HTTP-date | delta-seconds )

Two examples of its use are

       Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
       Retry-After: 120

Subtasks

Task #12442: Review retry-after-headers (libcloud fork)ResolvedPeter Amstutz

Task #12591: Review 12318-update-libcloud-pinClosedPeter Amstutz

Associated revisions

Revision dc003f39 (diff)
Added by Lucas Di Pentima almost 2 years ago

12318: Use libcloud fork 2.2.2.dev3

Includes a fix for delete_old_vhd, limit number of retries on
destroy_node & a fix on Retry-After header handling on
exceptions.

Refs #12318, #12452, #12324

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

Revision 3156c76f
Added by Lucas Di Pentima almost 2 years ago

Merge branch '12318-update-libcloud-pin'
Closes #12318

Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <>

History

#1 Updated by Tom Morris almost 2 years ago

  • Target version set to 2017-10-25 Sprint

#2 Updated by Lucas Di Pentima almost 2 years ago

  • Assigned To set to Lucas Di Pentima

#3 Updated by Lucas Di Pentima almost 2 years ago

  • Target version changed from 2017-10-25 Sprint to 2017-11-08 Sprint

#4 Updated by Lucas Di Pentima almost 2 years ago

  • Status changed from New to In Progress

#5 Updated by Lucas Di Pentima almost 2 years ago

PR Submitted: https://github.com/apache/libcloud/pull/1139 - branch retry-after-headers on our fork.

#6 Updated by Lucas Di Pentima almost 2 years ago

  • Target version changed from 2017-11-08 Sprint to 2017-11-22 Sprint

#7 Updated by Lucas Di Pentima almost 2 years ago

Finally PR 1139 was merged to libcloud trunk.

Update dc003f39b (branch 12318-update-libcloud-pin) has a new libcloud fork dependency (2.2.2.dev3 - https://github.com/curoverse/libcloud/tree/apache-libcloud-2.2.2.dev3) that includes all 3 pull requests: 1134, 1137 & 1139.

Test run: https://ci.curoverse.com/job/developer-run-tests/497/

#10 Updated by Peter Amstutz almost 2 years ago

This LGTM, thanks.

#11 Updated by Anonymous almost 2 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados|commit:3156c76f53b4142beb912cb57fd66e343c9e09d4.

Also available in: Atom PDF