Arvados

Feature #9066

Updated by Tom Clegg about 8 years ago

h2. Background 

 Currently, keepstore will accept hold an unbounded number of client requests, each one with a goroutine connections open and blocking until a memory buffer is available. If too many clients connect, keepstore will run out of file descriptors. This means it's unable to accept new connections, and even worse, it sabotages the requests at the front of the queue: keepstore can't open files on disk, or make connections to remote storage services. 

 When there are lots of queued requests, the ones at the back of the queue will eventually time out anyway; it would be better if those clients could find out right away that the server is overloaded. 

 h2. Proposed fix 

 Add a -max-requests -max-clients argument. When this many requests are already being handled/queued, If more clients connect than the configured limit, respond immediately to new requests with "503 Server Busy". 

 If the -max-requests -max-clients argument is not given, or is set to 0, use 2x max-buffers. 

 Note this applies to all requests, clients, even ones that don't wait for buffers (e.g., index and status). 

 h2. Implementation 

 Add a ConnectionLimiterFunc function to source:sdk/go/httpserver 

 <pre> 
 struct limiterHandler { 
          clients chan struct{} 
          handler http.Handler 
 } 

 func NewConnectionLimiter(maxClients int, handler http.Handler) { 
         return &limiterHandler{ 
                 clients: make(chan struct{}, maxClients), 
                 handler: handler, 
         } 
 } 

 func (h *limiterHandler) ServeHTTP(resp http.ResponseWriter, req *http.Request) { 
         select { 
         case h.clients <- struct{}{}: 
         default: 
                // reached max clients 
                resp.WriteHeader(http.StatusServiceUnavailable) 
                return 
         } 
         h.handler(resp, req) 
         <- clients 
 } 
 </pre> 

 In source:services/keepstore/logging_router.go, wrap MakeRESTRouter() with this handler. 

 Additional features, time permitting: 
 * Add a Len() method ("return len(h.clients)") to limiterHandler, and report that in keepstore's status.json 

Back