Project

General

Profile

Actions

Bug #12318

closed

Honor Retry-After headers on libcloud exceptions

Added by Lucas Di Pentima over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Node Manager
Target version:
Story points:
-

Description

It seems to be a bug on libcloud that could make nodemanager behave erratically on certain error situations with cloud providers.
As for libcloud 2.2.1, it seems that neither BaseHTTPError nor RateLimitReachedError get assigned the Retry-After header value, as it was removed from the exception_from_message() call on libcloud/libcloud/common/base.py.

Furthermore, the Retry-After header should accept a date in the future as detailed on https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html (relevant part copied below):

The Retry-After response-header field can be used with a 503 (Service Unavailable) response to indicate how long the service is expected to be unavailable to the requesting client. This field MAY also be used with any 3xx (Redirection) response to indicate the minimum time the user-agent is asked wait before issuing the redirected request. The value of this field can be either an HTTP-date or an integer number of seconds (in decimal) after the time of the response.

       Retry-After  = "Retry-After" ":" ( HTTP-date | delta-seconds )

Two examples of its use are

       Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
       Retry-After: 120

Subtasks 2 (0 open2 closed)

Task #12442: Review retry-after-headers (libcloud fork)ResolvedPeter Amstutz09/25/2017Actions
Task #12591: Review 12318-update-libcloud-pinClosedPeter Amstutz11/10/2017Actions
Actions #1

Updated by Tom Morris over 6 years ago

  • Target version set to 2017-10-25 Sprint
Actions #2

Updated by Lucas Di Pentima over 6 years ago

  • Assigned To set to Lucas Di Pentima
Actions #3

Updated by Lucas Di Pentima over 6 years ago

  • Target version changed from 2017-10-25 Sprint to 2017-11-08 Sprint
Actions #4

Updated by Lucas Di Pentima over 6 years ago

  • Status changed from New to In Progress
Actions #5

Updated by Lucas Di Pentima over 6 years ago

PR Submitted: https://github.com/apache/libcloud/pull/1139 - branch retry-after-headers on our fork.

Actions #6

Updated by Lucas Di Pentima over 6 years ago

  • Target version changed from 2017-11-08 Sprint to 2017-11-22 Sprint
Actions #7

Updated by Lucas Di Pentima over 6 years ago

Finally PR 1139 was merged to libcloud trunk.

Update dc003f39b (branch 12318-update-libcloud-pin) has a new libcloud fork dependency (2.2.2.dev3 - https://github.com/curoverse/libcloud/tree/apache-libcloud-2.2.2.dev3) that includes all 3 pull requests: 1134, 1137 & 1139.

Test run: https://ci.curoverse.com/job/developer-run-tests/497/

Actions #10

Updated by Peter Amstutz over 6 years ago

This LGTM, thanks.

Actions #11

Updated by Anonymous over 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados|commit:3156c76f53b4142beb912cb57fd66e343c9e09d4.

Actions

Also available in: Atom PDF