Feature #9066


[Keep] keepstore -max-requests limits concurrent requests

Added by Tom Clegg about 8 years ago. Updated about 8 years ago.

Assigned To:
Target version:
Story points:



Currently, keepstore will accept an unbounded number of client requests, each one with a goroutine blocking until a memory buffer is available. If too many clients connect, keepstore will run out of file descriptors. This means it's unable to accept new connections, and even worse, it sabotages the requests at the front of the queue: keepstore can't open files on disk, or make connections to remote storage services.

When there are lots of queued requests, the ones at the back of the queue will eventually time out anyway; it would be better if those clients could find out right away that the server is overloaded.

Proposed fix

Add a -max-requests argument. When this many requests are already being handled/queued, respond immediately to new requests with "503 Server Busy".

If the -max-requests argument is not given, or is set to 0, use 2x max-buffers.

Note this applies to all requests, even ones that don't wait for buffers (e.g., index and status).


Add a ConnectionLimiterFunc function to source:sdk/go/httpserver

struct limiterHandler {
         clients chan struct{}
         handler http.Handler

func NewConnectionLimiter(maxClients int, handler http.Handler) {
        return &limiterHandler{
                clients: make(chan struct{}, maxClients),
                handler: handler,

func (h *limiterHandler) ServeHTTP(resp http.ResponseWriter, req *http.Request) {
        select {
        case h.clients <- struct{}{}:
               // reached max clients
        h.handler(resp, req)
        <- clients

In source:services/keepstore/logging_router.go, wrap MakeRESTRouter() with this handler.

Additional features, time permitting:
  • Add a Len() method ("return len(h.clients)") to limiterHandler, and report that in keepstore's status.json

Subtasks 1 (0 open1 closed)

Task #9096: Review 9066-max-requestsResolvedRadhika Chippada04/26/2016Actions

Related issues

Related to Arvados - Bug #8825: [Keepstore] many connections in CLOSE_WAITResolved03/07/2017Actions
Actions #1

Updated by Tom Clegg about 8 years ago

  • Target version changed from Arvados Future Sprints to 2016-05-11 sprint
Actions #2

Updated by Tom Clegg about 8 years ago

  • Assigned To set to Tom Clegg
Actions #3

Updated by Tom Clegg about 8 years ago

  • Description updated (diff)
Actions #4

Updated by Tom Clegg about 8 years ago

  • Subject changed from [Keep] keepstore -max-clients limits concurrent connections to [Keep] keepstore -max-requests limits concurrent requests
Actions #5

Updated by Radhika Chippada about 8 years ago

  • Is the following a reasonable test scenario?
    • Create a request limiter of size 1
    • Send first request
    • While first request is in progress, send a second one (which gets 503)
    • Complete request 1
    • Send third request, which should get 200
Actions #6

Updated by Tom Clegg about 8 years ago

Yes, good point. Added to TestRequestLimiter1 in source:sdk/go/httpserver/request_limiter_test.go:

        if n200 != 1 || n503 != 9 {
                t.Fatalf("Got %d 200 responses, %d 503 responses (expected 1, 9)", n200, n503)
+       // Now that all 10 are finished, an 11th request should
+       // succeed.
+       go func() {
+               <-h.inHandler
+               h.okToProceed <- struct{}{}
+       }()
+       resp := httptest.NewRecorder()
+       l.ServeHTTP(resp, &http.Request{})
+       if resp.Code != 200 {
+               t.Errorf("Got status %d on 11th request, want 200", resp.Code)
+       }


Actions #7

Updated by Tom Clegg about 8 years ago

  • Status changed from New to In Progress
Actions #8

Updated by Tom Clegg about 8 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF