Bug #11136

Nginx config should speak JSON when returning its own response for unproxyable API requests

Added by Tom Clegg 8 months ago. Updated about 1 month ago.

Status:NewStart date:
Priority:NormalDue date:
Assignee:Nico César% Done:

0%

Category:-
Target version:Arvados Future Sprints
Story points1.0
Velocity based estimate0 days

Description

Also update the install documentation to explain to customers how to do this.


Related issues

Duplicated by Arvados - Bug #6651: "This website is under heavy load" error makes a parsing ... New 07/16/2015

History

#1 Updated by Tom Morris 8 months ago

  • Target version set to 2017-03-15 sprint

#2 Updated by Tom Morris 8 months ago

  • Description updated (diff)
  • Assignee set to Nico César

#3 Updated by Nico César 7 months ago

  • Target version changed from 2017-03-15 sprint to 2017-03-29 sprint

#4 Updated by Nico César 7 months ago

  • Story points set to 1.0

#5 Updated by Nico César 7 months ago

  • Target version changed from 2017-03-29 sprint to 2017-04-12 sprint

#6 Updated by Nico César 7 months ago

One approach is to do something like:

  error_page 503 @maintenance;
  location @maintenance {
    internal;
    if ($http_accept ~ json) {
      return 503 "{...}";
    }
    rewrite  ^(.*)$  /maintenance.html last;
    break;
  }

where "{...}" should have the json we return.

What should it be? because I imagine that we can have blacklisted endpoints or system down, or even an update going in the background.

Tom, can you elaborate where this ticket comes from and what we want to accomplish as you can see implementation is easy, we just need to come with consensus on the list of reasons and respective replies

#7 Updated by Tom Clegg 7 months ago

When apiserver itself responds ≥400, it returns a body like this

{"errors":["My hovercraft is full of eels"]}

We've seen several instances where client code has tried to parse the response body (either in an attempt to extract useful error messages like the one above and propagate them to the user, or just out of habit) and ended up dumping a JSON-parsing error and/or a chunk of HTML coming from Nginx instead. This tends to obscure the real problem.

Of course, clients should handle "no json" gracefully (e.g., there could be other proxies between client and server that we don't control) but in the meantime the proxy config should minimize the occurrence of the showing-raw-HTML behavior.

IIRC the most common/obvious cases are 502-504. It would be an improvement if these response bodies looked like

HTTP/1.1 502 Bad gateway
Content-type: application/json

{"errors":["Invalid API response from upstream server"]}
HTTP/1.1 503 Service Unavailable
Content-type: application/json

{"errors":["API temporarily unavailable"]}
HTTP/1.1 504 Gateway timeout
Content-type: application/json

{"errors":["API request timed out"]}

We should return JSON by default even if the request doesn't say "Accept: application/json". If we are eager to return "friendly" html error pages to browsers, we should do that only when the request's Accept header specifically mentions text/html.

#8 Updated by Nico César 6 months ago

  • Target version changed from 2017-04-12 sprint to 2017-04-26 sprint

#9 Updated by Tom Morris 6 months ago

  • Target version changed from 2017-04-26 sprint to 2017-05-10 sprint

#10 Updated by Nico César 5 months ago

  • Target version changed from 2017-05-10 sprint to 2017-05-24 sprint

#11 Updated by Nico César 5 months ago

  • Target version changed from 2017-05-24 sprint to 2017-06-07 sprint

#12 Updated by Nico César 4 months ago

  • Target version changed from 2017-06-07 sprint to 2017-06-21 sprint

#13 Updated by Nico César 4 months ago

  • Target version changed from 2017-06-21 sprint to 2017-07-05 sprint

#14 Updated by Nico César 3 months ago

  • Target version changed from 2017-07-05 sprint to 2017-07-19 sprint

#15 Updated by Nico César 3 months ago

  • Target version changed from 2017-07-19 sprint to 2017-08-02 sprint

#16 Updated by Nico César 3 months ago

  • Target version changed from 2017-08-02 sprint to 2017-08-16 sprint

#17 Updated by Tom Morris 2 months ago

The linked #6651 contains one recipe for fixing this.

#18 Updated by Nico César 2 months ago

  • Target version changed from 2017-08-16 sprint to 2017-08-30 Sprint

#19 Updated by Tom Morris about 1 month ago

  • Target version changed from 2017-08-30 Sprint to Arvados Future Sprints

Also available in: Atom PDF