Bug #9427
Updated by Peter Amstutz almost 8 years ago
The websockets server currently uses the EventMachine library for multiplexing events. This is a single threaded event queue (despite the fact that we use it with Puma, which is a thread-per-connection web server).
We're seeing severe lag, where new connections respond slowly or not at all. This seems to be correlated if any connection is using the "catch up" functionality to replay past events. Upon instrumenting the server, we can see the performance:
<pre>
2016-06-16_20:30:05.76986 #<Faye::WebSocket:0x000000049298d0> sending 28643742
2016-06-16_20:30:05.87824 #<Faye::WebSocket:0x000000049298d0> sent 28643742
2016-06-16_20:30:05.87832 #<Faye::WebSocket:0x000000049298d0> sending 28643743
2016-06-16_20:30:05.99874 #<Faye::WebSocket:0x000000049298d0> sent 28643743
2016-06-16_20:30:05.99882 #<Faye::WebSocket:0x000000049298d0> sending 28643745
2016-06-16_20:30:06.10723 #<Faye::WebSocket:0x000000049298d0> sent 28643745
2016-06-16_20:30:06.10731 #<Faye::WebSocket:0x000000049298d0> sending 28643746
2016-06-16_20:30:06.21399 #<Faye::WebSocket:0x000000049298d0> sent 28643746
2016-06-16_20:30:06.21407 #<Faye::WebSocket:0x000000049298d0> sending 28643749
2016-06-16_20:30:06.32665 #<Faye::WebSocket:0x000000049298d0> sent 28643749
2016-06-16_20:30:06.32675 #<Faye::WebSocket:0x000000049298d0> sending 28643750
2016-06-16_20:30:06.43588 #<Faye::WebSocket:0x000000049298d0> sent 28643750
</pre>
In this trace it is spending about 100ms per invocation of websocket.send(). This is probably due to a slow or backlogged client connection. Because all events are currently processed in a single thread, this means the websocket server can't scale past the speed of the slowest client.
This may also be affected by a related performance problem: every event has to run through the permission checks to check if the user is allowed to read the event; as the number of groups readable by a user becomes large, this check becomes expensive.
Proposed solution:
Create a separate event machine queue per thread, so that slow connections won't interfere with other connections.