Arvados

h1. Websocket server

(early draft)

{{toc}}

h2. Background

The Rails API server can function as a websocket server. Clients (notably Workbench, arv-mount, arv-ws) use it to listen for events without polling.

Problems with current implementation:

* Unreliable. See #9427, #8277

* Resource-heavy (one postgres connection per connected client, uses lots of memory)

* Logging is not very good

* Updates look like database records instead of API responses (e.g., computed fields are missing, collection manifest_text has no signatures)

* Offers an API for catching up on missed events after disconnecting/reconnecting, but this API (let alone the code) isn't enough to offer a "don't miss any events, don't send any events twice" guarantee. See #9388

#8460

h2. Desired features

Monotonically increasing event IDs, so clients can (meaningfully) request "all matching events since X"

h2. Design sketch (TC)

New server, written in Go.

One goroutine per connected client.

One database connection receiving notifications about new logs. (Possibly still N database connections serving "catch-up" messages to N clients.)

h2. Design sketch (PA)

When client connects, it can request a new session (with event filter), or asks to resume an existing session from a given event id.

Each session has a session id and is associated with a user, an event channel, an event queue, event filters, and exactly one websocket connection.

Clients can create multiple sessions on the same websocket.  When a session is created, the client can request a replay of events from database, either "last N" or "since time T".

Resuming a session associates the session to a new websocket connection (must be same user).  Resuming a session replays any queued events occurring after the given event id.

Orderly websocket disconnects tear down any associated sessions.  Abrupt disconnects maintain the sessions for N minutes.  Clients can also end individual sessions without disconnecting the websocket.

NOTIFY sends full json-encoded record.

Websocket server database LISTEN receives NOTIFY, deserializes record, assigns a sequence number to the event, gets @object_uuid@ and @object_owner_uuid@ and determines the set of users who have permission to view the log record using in-memory cache of permissions graph.  

Intersect the set of users who can view the record with the users associated with active sessions and adds the record to the queue for each session with the associated user.

The goroutine for session gets a new event on the event channel.  It first applies the session filters to determine if the event should be propagated.  (This should be a Go-based implementation of Arvados query filters and not touch the database.)  If it passes filters it is added to a queue, and can be sent to the websocket client.

The websocket client receives the event and sends an acknowledgement with the sequence number.  The server receives the acknowledgement and removes messages from the queue with sequence number less than or equal to the acknowledgement.

h2. Libraries

Websocket:

* https://godoc.org/golang.org/x/net/websocket

PostgreSQL:

* https://godoc.org/github.com/lib/pq via https://godoc.org/database/sql

* https://godoc.org/github.com/lib/pq#hdr-Notifications and https://godoc.org/github.com/lib/pq/listen_example

h2. Obstacles

#8565, #8566

h2. Related

It might be expedient to offload synchronization to some existing software that does this well.

* "Apache Zookeeper":https://zookeeper.apache.org/doc/trunk/zookeeperOver.html -- "Coordination services are notoriously hard to get right. They are especially prone to errors such as race conditions and deadlock. The motivation behind ZooKeeper is to relieve distributed applications the responsibility of implementing coordination services from scratch."

* Google "Chubby":http://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf paper

* NSQ http://nsq.io/