Bug #15721

OPTIONS on /arvados/v1/config returns status 503

Added by Peter Amstutz 6 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/15/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Requesting OPTIONS on /arvados/v1/config returns a 503 status code:

$ curl -v -X OPTIONS https://c97qk.arvadosapi.com/arvados/v1/config
...
< HTTP/1.1 503 Service Unavailable
< Server: nginx/1.15.8
< Date: Fri, 11 Oct 2019 20:22:24 GMT
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Headers: Authorization, Content-Type
< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 86486400
< 
* Curl_http_done: called premature == 0
* Connection #0 to host c97qk.arvadosapi.com left intact

This is a bug in master, since it works on 1.4.1:

$ curl -v -X OPTIONS https://qr1hi.arvadosapi.com/arvados/v1/config
...
< HTTP/1.1 200 OK
< Server: nginx/1.15.8
< Date: Fri, 11 Oct 2019 20:23:45 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Access-Control-Allow-Headers: Authorization, Content-Type
< Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE
< Access-Control-Allow-Origin: *
< Access-Control-Max-Age: 86486400
< Cache-Control: no-cache
< Status: 200 OK
< Strict-Transport-Security: max-age=15552000
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Powered-By: Phusion Passenger 6.0.2
< X-Request-Id: req-15o843ywr0rr16tit4ws
< X-Runtime: 0.003235
< X-Xss-Protection: 1; mode=block
< 
* Curl_http_done: called premature == 0
* Connection #0 to host qr1hi.arvadosapi.com left intact

This is a blocker because it breaks workbench 2 multi-site search (found while working on #15530)


Subtasks

Task #15723: Review 15721-config-corsResolvedPeter Amstutz

Associated revisions

Revision 2a3a0f7b
Added by Tom Clegg 6 months ago

Merge branch '15721-config-cors'

fixes #15721

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz 6 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 6 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 6 months ago

  • Status changed from In Progress to New

#4 Updated by Peter Amstutz 6 months ago

  • Description updated (diff)
  • Priority changed from Normal to High

#5 Updated by Tom Clegg 6 months ago

Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...

#6 Updated by Tom Clegg 6 months ago

  • Assigned To set to Tom Clegg

#7 Updated by Peter Amstutz 6 months ago

Tom Clegg wrote:

Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...

It might be sending an authorization header or something that forces a preflight request. I didn't look too closely since this is something that used to work, I don't think workbench2 is doing anything wrong.

#8 Updated by Tom Clegg 6 months ago

Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:

2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)

15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/

If you need a workaround -- configure MaxConcurrentRequests: 1000

#9 Updated by Tom Clegg 6 months ago

  • Status changed from New to In Progress

#10 Updated by Peter Amstutz 6 months ago

Tom Clegg wrote:

Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:

2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)

15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/

LGTM

If you need a workaround -- configure MaxConcurrentRequests: 1000

We could also set a default value for MaxConcurrentRequests and require that it be > 0, seems like one of those things where you always want a limit to mitigate accidental or deliberate denial of service.

#11 Updated by Tom Clegg 6 months ago

Changing how we spell "unlimited" from "0" to "99999999" (for the sake of reserving "0" to mean "refuse to start") doesn't sound appealing to me. I don't object to changing the default although it doesn't feel like a priority.

#12 Updated by Tom Clegg 6 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

#13 Updated by Peter Amstutz 3 months ago

  • Release set to 22

Also available in: Atom PDF