Arvados

Option 1 sounds pretty good.

Without some safeguard, a client that receives too slowly would grow its outbound queue until the server goes OOM. Should we deal with this by disconnecting any client whose server-side queue has exceeded some configurable limit?

Can we also check the queue size when we send to the queue, instead of just when we receive from it? Once the queue size is 1000, there's no point adding anything else to it, because next time the client thread wakes up it will throw everything away and disconnect.

Max queue size should be configurable.

Should probably exit the socket event loop after sending the 500 error. (If I'm following correctly, current code will try to send a 500 error for each queued event until the queue size drops below threshold, which I assume is not what you intended to do after closing the socket...)

Max connection limit should be configurable.

Is it possible any of this is testable?

With the caveat that it's feeling very friday-5pmish right now... other than the above points this LGTM. It will be great to get out from under this "client A has to wait for client B's slow network" nonsense. Thanks.

9427-threaded-websockets @ 27b18bf

• Adds configuration variables websocket_max_connections, websocket_max_notify_backlog, websocket_max_filters, and websocket_db_pool
• Adds initializer to adjust database connection pool size when running multithreaded websockets
• Check backlog size and close backlogged connections when pushing, not popping
• Increase wait time for tests (seems like there is some delay on initial websocket subscription, this may be due to creating new database connections).

I'm not keen on the amount of reaching into ActiveRecord's guts for the sake of having a different pool size depending on whether websocket server is enabled. If someone really wants different pool sizes running from the same config files, shouldn't they just put this in database.yml?

production:
  adapter: postgresql
  ...
  pool: 128

websocket:
  adapter: postgresql
  ...
  pool: 256

At the very least, the default in application.default.yml should be zero and signify "just use what I've configured in database.yml". But it would be even better if we could avoid the "application.yml overrides database.yml" situation entirely.

Tom Clegg wrote:

I'm not keen on the amount of reaching into ActiveRecord's guts for the sake of having a different pool size depending on whether websocket server is enabled. If someone really wants different pool sizes running from the same config files, shouldn't they just put this in database.yml?

[...]

At the very least, the default in application.default.yml should be zero and signify "just use what I've configured in database.yml". But it would be even better if we could avoid the "application.yml overrides database.yml" situation entirely.

Since apparently "pool" in database.yml does work, backed this and added a comment to database.yml.example instead. Now @ fa615bf

LGTM, thanks!