Bug #9427
Updated by Peter Amstutz about 8 years ago
The websockets server currently uses the EventMachine library for multiplexing events. This is a single threaded event queue (despite the fact that we use it with Puma, which is a thread-per-connection web server). We're seeing severe lag, where new connections respond slowly or not at all. This seems to be correlated if any connection is using the "catch up" functionality to replay past events. Upon instrumenting the server, we can see the performance: <pre> 2016-06-16_20:30:05.76986 #<Faye::WebSocket:0x000000049298d0> sending 28643742 2016-06-16_20:30:05.87824 #<Faye::WebSocket:0x000000049298d0> sent 28643742 2016-06-16_20:30:05.87832 #<Faye::WebSocket:0x000000049298d0> sending 28643743 2016-06-16_20:30:05.99874 #<Faye::WebSocket:0x000000049298d0> sent 28643743 2016-06-16_20:30:05.99882 #<Faye::WebSocket:0x000000049298d0> sending 28643745 2016-06-16_20:30:06.10723 #<Faye::WebSocket:0x000000049298d0> sent 28643745 2016-06-16_20:30:06.10731 #<Faye::WebSocket:0x000000049298d0> sending 28643746 2016-06-16_20:30:06.21399 #<Faye::WebSocket:0x000000049298d0> sent 28643746 2016-06-16_20:30:06.21407 #<Faye::WebSocket:0x000000049298d0> sending 28643749 2016-06-16_20:30:06.32665 #<Faye::WebSocket:0x000000049298d0> sent 28643749 2016-06-16_20:30:06.32675 #<Faye::WebSocket:0x000000049298d0> sending 28643750 2016-06-16_20:30:06.43588 #<Faye::WebSocket:0x000000049298d0> sent 28643750 </pre> In this trace it is spending about 100ms per invocation of websocket.send(). This is probably due to a slow or backlogged client connection. Because all events are currently processed in a single thread, this means the websocket server can't scale past the speed of the slowest client. This may also be affected by a related performance problem: every event has to run through the permission checks to check if the user is allowed to read the event; as the number of groups readable by a user becomes large, this check becomes expensive. Proposed solution: We need a thread per connection for blocking operations (accessing the database, sending events). It would be nice if we could create an EventMachine "reactor" per thread, however this doesn't appear to be possible. EventMachine has thread pool functionality with "defer". We may able to make use of that instead of @next_tick@.