Project

General

Profile

Actions

Bug #15721

closed

OPTIONS on /arvados/v1/config returns status 503

Added by Peter Amstutz over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/15/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Requesting OPTIONS on /arvados/v1/config returns a 503 status code:

$ curl -v -X OPTIONS https://c97qk.arvadosapi.com/arvados/v1/config
...
< HTTP/1.1 503 Service Unavailable
< Server: nginx/1.15.8
< Date: Fri, 11 Oct 2019 20:22:24 GMT
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Headers: Authorization, Content-Type
< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 86486400
< 
* Curl_http_done: called premature == 0
* Connection #0 to host c97qk.arvadosapi.com left intact

This is a bug in master, since it works on 1.4.1:

$ curl -v -X OPTIONS https://qr1hi.arvadosapi.com/arvados/v1/config
...
< HTTP/1.1 200 OK
< Server: nginx/1.15.8
< Date: Fri, 11 Oct 2019 20:23:45 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Headers: Authorization, Content-Type
< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 86486400
< Cache-Control: no-cache
< Status: 200 OK
< Strict-Transport-Security: max-age=15552000
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Powered-By: Phusion Passenger 6.0.2
< X-Request-Id: req-15o843ywr0rr16tit4ws
< X-Runtime: 0.003235
< X-Xss-Protection: 1; mode=block
< 
* Curl_http_done: called premature == 0
* Connection #0 to host qr1hi.arvadosapi.com left intact

This is a blocker because it breaks workbench 2 multi-site search (found while working on #15530)


Subtasks 1 (0 open1 closed)

Task #15723: Review 15721-config-corsResolvedPeter Amstutz10/15/2019

Actions
Actions #1

Updated by Peter Amstutz over 4 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz over 4 years ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz over 4 years ago

  • Status changed from In Progress to New
Actions #4

Updated by Peter Amstutz over 4 years ago

  • Description updated (diff)
  • Priority changed from Normal to High
Actions #5

Updated by Tom Clegg over 4 years ago

Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...

Actions #6

Updated by Tom Clegg over 4 years ago

  • Assigned To set to Tom Clegg
Actions #7

Updated by Peter Amstutz over 4 years ago

Tom Clegg wrote:

Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...

It might be sending an authorization header or something that forces a preflight request. I didn't look too closely since this is something that used to work, I don't think workbench2 is doing anything wrong.

Actions #8

Updated by Tom Clegg over 4 years ago

Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:

2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)

15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/

If you need a workaround -- configure MaxConcurrentRequests: 1000

Actions #9

Updated by Tom Clegg over 4 years ago

  • Status changed from New to In Progress
Actions #10

Updated by Peter Amstutz over 4 years ago

Tom Clegg wrote:

Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:

2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)

15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/

LGTM

If you need a workaround -- configure MaxConcurrentRequests: 1000

We could also set a default value for MaxConcurrentRequests and require that it be > 0, seems like one of those things where you always want a limit to mitigate accidental or deliberate denial of service.

Actions #11

Updated by Tom Clegg over 4 years ago

Changing how we spell "unlimited" from "0" to "99999999" (for the sake of reserving "0" to mean "refuse to start") doesn't sound appealing to me. I don't object to changing the default although it doesn't feel like a priority.

Actions #12

Updated by Tom Clegg over 4 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100
Actions #13

Updated by Peter Amstutz about 4 years ago

  • Release set to 22
Actions

Also available in: Atom PDF