Arvados

h1. Message queue

Arvados needs a (better) message-passing facility for system components to use.

h2. Motivation / background

Sometimes Arvados clients and system components need to pass messages to one another. Often, REST makes sense. But in some cases (e.g., it's not practical for the sender to know the message destination(s), or it's not efficient for the sender to initiate an HTTP request to each destination) something more like pubsub is needed.

Currently (2017) we use PostgreSQL notify/listen as a pubsub device. This is very inefficient, primarily because Postgresql messages are very short. We pass longer messages by inserting the message to a "logs" table on disk, and sending the resulting ID to subscribers, who then select the real message. Eventually (depending on server config) we delete old rows.

h2. Components/features that need a message queue

* showing live container logs in workbench/terminal

* crunch stderr logs

* system logs

* container added to queue → noticed by dispatch

* container cancelled → noticed by dispatch

* container state/progress has changed, workbench should update page

* container finished, here is the output (maybe REST API is still best for this)

* get current state of object and notify me when it changes relative to that

* arvados-ws (“cache invalidation”) -- workbench, fuse

* (future) "latest state of object" service, similar to arvados-ws but more convenient for the client

* permission graph is updated (?)

h2. Desired capabilities of a message queue

* Support for WebSocket transport to enable use by browser

* Client support for languages used in Arvados (Go, Python, Ruby, Javascript)

* Can apply Arvados permission model

* Pub/sub of live change events on records

** Transactional subscribe request that returns most recent record + subscribes to subsequent changes

* Pub/sub of live Container logs

** Transactional subscribe request that returns recent recent log history + subscribes to subsequent logs

** In-order delivery of logs

* Handle temporary disconnects

* Easy to deploy (fits into our stack)

* Scales to at least 1000 streams without trouble

Investigate WAMP:

http://wamp-proto.org/

It uses WebSockets as its principal transport, and has client library implementations for the languages we care about (Go, Python, Javascript)

https://github.com/gammazero/nexus

https://github.com/crossbario/autobahn-python

https://github.com/crossbario/autobahn-js

Probably what we want to do is run a "WAMP router" that sits in the middle of the log producers, the logging microservice, and log listeners (browser).