Story #13697

Prevent the API server and database from continuing to serve requests to clients after timeout

Added by Joshua Randall about 1 year ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
API
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
0.5

Description

A typical nginx configuration for the API server is to time out requests after 5m (300s). However, passenger/rails currently do not notice that the client has given up on the request and continue to process it. On some occasions (when things go wrong in other ways) we have had requests running for hours, holding locks that prevent other parts of the system from working, and in the worst cases, effecting a DoS on the entire API server by consuming all available passenger workers in the process.

There is no point in processing any part of a request for longer than the request timeout. An easy win to prevent the above scenario would be to set a postgres statement timeout to the same length as the nginx gateway timeout.

Our system has been running with a 300s statement_timeout for a few weeks without issue: https://github.com/wtsi-hgi/arvados/commit/d9728e17148db53caf1f16fce032448c3d5c1432

Probably the value used for the timeout should come from config rather than being hard-coded, so that admins can configure it appropriately when non-standard nginx configuration is used.

History

#1 Updated by Tom Morris 5 months ago

  • Target version set to To Be Groomed

#2 Updated by Tom Morris 5 months ago

  • Story points set to 0.5

#3 Updated by Tom Morris 4 months ago

  • Target version changed from To Be Groomed to Arvados Future Sprints

Also available in: Atom PDF