Story #13697

Prevent the API server and database from continuing to serve requests to clients after timeout

Added by Joshua Randall about 3 years ago. Updated 21 days ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


A typical nginx configuration for the API server is to time out requests after 5m (300s). However, passenger/rails currently do not notice that the client has given up on the request and continue to process it. On some occasions (when things go wrong in other ways) we have had requests running for hours, holding locks that prevent other parts of the system from working, and in the worst cases, effecting a DoS on the entire API server by consuming all available passenger workers in the process.

There is no point in processing any part of a request for longer than the request timeout. An easy win to prevent the above scenario would be to set a postgres statement timeout to the same length as the nginx gateway timeout.

Our system has been running with a 300s statement_timeout for a few weeks without issue:

Probably the value used for the timeout should come from config rather than being hard-coded, so that admins can configure it appropriately when non-standard nginx configuration is used.


#1 Updated by Tom Morris over 2 years ago

  • Target version set to To Be Groomed

#2 Updated by Tom Morris over 2 years ago

  • Story points set to 0.5

#3 Updated by Tom Morris over 2 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints

#4 Updated by Peter Amstutz 21 days ago

  • Target version deleted (Arvados Future Sprints)

Also available in: Atom PDF