Bug #15721
closed
OPTIONS on /arvados/v1/config returns status 503
Description
Requesting OPTIONS on /arvados/v1/config returns a 503 status code:
$ curl -v -X OPTIONS https://c97qk.arvadosapi.com/arvados/v1/config ... < HTTP/1.1 503 Service Unavailable < Server: nginx/1.15.8 < Date: Fri, 11 Oct 2019 20:22:24 GMT < Content-Length: 0 < Connection: keep-alive < Access-Control-Allow-Headers: Authorization, Content-Type < Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE < Access-Control-Allow-Origin: * < Access-Control-Max-Age: 86486400 < * Curl_http_done: called premature == 0 * Connection #0 to host c97qk.arvadosapi.com left intact
This is a bug in master, since it works on 1.4.1:
$ curl -v -X OPTIONS https://qr1hi.arvadosapi.com/arvados/v1/config ... < HTTP/1.1 200 OK < Server: nginx/1.15.8 < Date: Fri, 11 Oct 2019 20:23:45 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 0 < Connection: keep-alive < Access-Control-Allow-Headers: Authorization, Content-Type < Access-Control-Allow-Methods: GET, HEAD, PUT, POST, DELETE < Access-Control-Allow-Origin: * < Access-Control-Max-Age: 86486400 < Cache-Control: no-cache < Status: 200 OK < Strict-Transport-Security: max-age=15552000 < X-Content-Type-Options: nosniff < X-Frame-Options: SAMEORIGIN < X-Powered-By: Phusion Passenger 6.0.2 < X-Request-Id: req-15o843ywr0rr16tit4ws < X-Runtime: 0.003235 < X-Xss-Protection: 1; mode=block < * Curl_http_done: called premature == 0 * Connection #0 to host qr1hi.arvadosapi.com left intact
This is a blocker because it breaks workbench 2 multi-site search (found while working on #15530)
Updated by Peter Amstutz over 4 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz over 4 years ago
- Status changed from In Progress to New
Updated by Peter Amstutz over 4 years ago
- Description updated (diff)
- Priority changed from Normal to High
Updated by Tom Clegg over 4 years ago
Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...
Updated by Peter Amstutz over 4 years ago
Tom Clegg wrote:
Hm. I wouldn't expect OPTIONS to return 503 -- but I wouldn't expect an OPTIONS (preflight) request for the config endpoint at all, since the actual request is a GET...
It might be sending an authorization header or something that forces a preflight request. I didn't look too closely since this is something that used to work, I don't think workbench2 is doing anything wrong.
Updated by Tom Clegg over 4 years ago
Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:
2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/
If you need a workaround -- configure MaxConcurrentRequests: 1000
Updated by Peter Amstutz over 4 years ago
Tom Clegg wrote:
Missing "return" statement. In the "no request limit configured" case we were calling the handler, then returning 503 because we'd already reached the maximum of 0 concurrent requests. Same bug was also causing a lot of this:
2019/10/15 14:10:18 http: superfluous response.WriteHeader call from git.curoverse.com/arvados.git/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:59)
15721-config-cors @ ef25437d74cd7c2ce9ba4df2364c1da3d06b8bbb -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1585/
LGTM
If you need a workaround -- configure MaxConcurrentRequests: 1000
We could also set a default value for MaxConcurrentRequests and require that it be > 0, seems like one of those things where you always want a limit to mitigate accidental or deliberate denial of service.
Updated by Tom Clegg over 4 years ago
Changing how we spell "unlimited" from "0" to "99999999" (for the sake of reserving "0" to mean "refuse to start") doesn't sound appealing to me. I don't object to changing the default although it doesn't feel like a priority.
Updated by Tom Clegg over 4 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|2a3a0f7b34e43e436151517c7c262f43af674cbf.