Feature #9066

[Keep] keepstore -max-requests limits concurrent requests

Added by Tom Clegg over 5 years ago. Updated over 5 years ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)
Story points:



Currently, keepstore will accept an unbounded number of client requests, each one with a goroutine blocking until a memory buffer is available. If too many clients connect, keepstore will run out of file descriptors. This means it's unable to accept new connections, and even worse, it sabotages the requests at the front of the queue: keepstore can't open files on disk, or make connections to remote storage services.

When there are lots of queued requests, the ones at the back of the queue will eventually time out anyway; it would be better if those clients could find out right away that the server is overloaded.

Proposed fix

Add a -max-requests argument. When this many requests are already being handled/queued, respond immediately to new requests with "503 Server Busy".

If the -max-requests argument is not given, or is set to 0, use 2x max-buffers.

Note this applies to all requests, even ones that don't wait for buffers (e.g., index and status).


Add a ConnectionLimiterFunc function to source:sdk/go/httpserver

struct limiterHandler {
         clients chan struct{}
         handler http.Handler

func NewConnectionLimiter(maxClients int, handler http.Handler) {
        return &limiterHandler{
                clients: make(chan struct{}, maxClients),
                handler: handler,

func (h *limiterHandler) ServeHTTP(resp http.ResponseWriter, req *http.Request) {
        select {
        case h.clients <- struct{}{}:
               // reached max clients
        h.handler(resp, req)
        <- clients

In source:services/keepstore/logging_router.go, wrap MakeRESTRouter() with this handler.

Additional features, time permitting:
  • Add a Len() method ("return len(h.clients)") to limiterHandler, and report that in keepstore's status.json


Task #9096: Review 9066-max-requestsResolvedRadhika Chippada

Related issues

Related to Arvados - Bug #8825: [Keepstore] many connections in CLOSE_WAITResolved03/07/2017

Associated revisions

Revision 1535c8d1
Added by Tom Clegg over 5 years ago

Merge branch '9066-max-requests'

refs #9066


#1 Updated by Tom Clegg over 5 years ago

  • Target version changed from Arvados Future Sprints to 2016-05-11 sprint

#2 Updated by Tom Clegg over 5 years ago

  • Assigned To set to Tom Clegg

#3 Updated by Tom Clegg over 5 years ago

  • Description updated (diff)

#4 Updated by Tom Clegg over 5 years ago

  • Subject changed from [Keep] keepstore -max-clients limits concurrent connections to [Keep] keepstore -max-requests limits concurrent requests

#5 Updated by Radhika Chippada over 5 years ago

  • Is the following a reasonable test scenario?
    • Create a request limiter of size 1
    • Send first request
    • While first request is in progress, send a second one (which gets 503)
    • Complete request 1
    • Send third request, which should get 200

#6 Updated by Tom Clegg over 5 years ago

Yes, good point. Added to TestRequestLimiter1 in source:sdk/go/httpserver/request_limiter_test.go:

        if n200 != 1 || n503 != 9 {
                t.Fatalf("Got %d 200 responses, %d 503 responses (expected 1, 9)", n200, n503)
+       // Now that all 10 are finished, an 11th request should
+       // succeed.
+       go func() {
+               <-h.inHandler
+               h.okToProceed <- struct{}{}
+       }()
+       resp := httptest.NewRecorder()
+       l.ServeHTTP(resp, &http.Request{})
+       if resp.Code != 200 {
+               t.Errorf("Got status %d on 11th request, want 200", resp.Code)
+       }


#7 Updated by Tom Clegg over 5 years ago

  • Status changed from New to In Progress

#8 Updated by Tom Clegg over 5 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF