Bug #11537
closed
[keepproxy] detect proxy loop (broken config) -- return an error if keepproxy receives a request from itself
Added by Tom Clegg almost 8 years ago.
Updated over 7 years ago.
Description
The manual setup procedure makes it easy to misconfigure keepproxy and Nginx such that keepproxy fulfills requests by forwarding requests to itself. Even when setup/configuration is automated, bugs and races could cause this condition sometimes.
Keepproxy should detect this situation and return an error and log an appropriate message, to make it easier for a sysadmin to detect.
The HTTP "Via" header is good for this. See https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.45
Implementation:
- in sdk/go/keepclient, add a field "Do func(*http.Request) (*http.Response, error)".
- if kc.Do is not nil, keepclient should use that function instead of instead of kc.Client.Do().
- keepproxy should set kc.Do to a func that adds a Via header (e.g., "Via: HTTP/1.1 keepproxy" where HTTP/1.1 is req.Proto) and then calls kc.Client.Do().
- if an incoming request has a Via header containing the string " keepproxy", respond 500 and log an error message ("proxy loop detected: perhaps keepproxy is misidentified by gateway config as an external client, or its keep_services record does not have service_type=proxy?")
- Subject changed from [keepproxy] return an error if keepproxy receives a request from itself to [keepproxy] detect proxy loop (broken config) -- return an error if keepproxy receives a request from itself
- Target version set to 2017-05-10 sprint
- Story points set to 1.0
- Assigned To set to Tom Clegg
- Status changed from New to In Progress
Implementation:
- Instead of adding a Do field to keepclient.KeepClient as described, I changed the Client field's type from *http.Client to a HTTPClient interface. Otherwise, using the Do field would make the Client field meaningless, except that the "set appropriate timeout according to proxy/disk services" code would edit it uselessly.
- In order to make the "loop detected" error visible to the outer keepproxy on writes, I had to adjust the keepclient error handling. Now, instead of just returning a generic "insufficient replicas" error object, it returns the errors returned by the backend servers, like we do for reads. I also updated the error identifiers to match Go conventions ("FooError" is a type, and "ErrFoo" is an object).
Functional:
11537-keepproxy-loop @ baa1c3db1f635828dda3cebdc47d98b33bb6518d
11537-keepproxy-loop LGTM
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:e56ae6aad06c37d5512537047871d7363dd97620.
Also available in: Atom
PDF