Events API » History » Version 3

Tom Clegg, 10/25/2016 06:53 AM

1 1 Tom Clegg
h1. Events API
2 1 Tom Clegg
3 1 Tom Clegg
(draft)
4 1 Tom Clegg
5 1 Tom Clegg
{{toc}}
6 1 Tom Clegg
7 1 Tom Clegg
See also: [[Websocket server]]
8 1 Tom Clegg
9 1 Tom Clegg
h1. Purpose
10 1 Tom Clegg
11 1 Tom Clegg
The Events API serves to notify processes about events that interest them _as soon as possible after those events happen_.
12 1 Tom Clegg
13 1 Tom Clegg
(The history of events that have happened in the past is also interesting, but that's addressed by the Logs API, not the Events API.)
14 1 Tom Clegg
15 1 Tom Clegg
h1. Conceptual model
16 1 Tom Clegg
17 1 Tom Clegg
An event reports a change to the state of an object.
18 1 Tom Clegg
19 1 Tom Clegg
The fact that an object's state has changed is meaningful only when its previous state is known. For example, if a client asks "tell me the next time object X changes" at nearly the same time X changes, the response depends on whether the request arrives before or after the change occurs.
20 1 Tom Clegg
21 1 Tom Clegg
Therefore, the Events API should support operations like:
22 1 Tom Clegg
* "tell me the current state of X, and then notify me next time it changes"
23 1 Tom Clegg
* "tell me as soon as X differs from my cached copy that has Etag E"
24 1 Tom Clegg
25 1 Tom Clegg
An "event stream" is a sequence of events about an object, starting from an implicit or explicit known state.
26 1 Tom Clegg
27 1 Tom Clegg
h1. Essential features
28 1 Tom Clegg
29 1 Tom Clegg
h2. Multiple streams
30 1 Tom Clegg
31 1 Tom Clegg
The Events API supports multiplexing event streams on a single connection. The cost of setting up and maintaining an event channel can be non-trivial, and the sequence of events concerning multiple related objects may be significant.
32 1 Tom Clegg
33 1 Tom Clegg
It is possible to add and remove event streams on an existing connection, without interrupting other streams.
34 1 Tom Clegg
35 1 Tom Clegg
It is permitted to hold a connection open with no event streams, but the server may close such connections after some time threshold.
36 1 Tom Clegg
37 1 Tom Clegg
h2. Delivery guarantees
38 1 Tom Clegg
39 1 Tom Clegg
In general, the Events API cannot guarantee that every event will be delivered.
40 1 Tom Clegg
41 3 Tom Clegg
However, there are specific cases where it is beneficial (and practical) to detect missed events and notify the client.
42 1 Tom Clegg
43 3 Tom Clegg
If some events are dropped but the event stream is still open (for example, a server-side buffer overflows because a client is receiving data too slowly) the server must indicate this to the client no later than the next event delivery.
44 3 Tom Clegg
* The "missed events" signal may specify a single event stream (UUID); if not, the client must interpret this as "events may have been missed on all active streams".
45 1 Tom Clegg
* The "missed events" signal does not necessarily specify the number of missed events.
46 1 Tom Clegg
* The server is permitted to send a "missed events" signal even if no events were missed.
47 1 Tom Clegg
48 1 Tom Clegg
Depending on the application, a client might respond to a "missed events" signal by
49 1 Tom Clegg
* restarting the affected streams immediately
50 1 Tom Clegg
* restarting the affected streams only if they stay silent for some timeout period
51 1 Tom Clegg
* doing nothing
52 1 Tom Clegg
* hanging up
53 1 Tom Clegg
54 1 Tom Clegg
h2. Event message content
55 1 Tom Clegg
56 3 Tom Clegg
Each event includes, at minimum, the UUID and Etag of the changed object.
57 1 Tom Clegg
58 3 Tom Clegg
h2. Non-state-changing events (logs)
59 1 Tom Clegg
60 3 Tom Clegg
Container/job log messages (e.g., stderr) should be available through the Events API, even though they don't correspond to an etag change in any object.
61 3 Tom Clegg
62 3 Tom Clegg
Given that the etag does not change, the client is obviously interested in other attributes of the event itself (e.g., stderr text), so those attributes must be
63 3 Tom Clegg
* included with the event payload, or
64 3 Tom Clegg
* stored in a Log object whose UUID is included in the event payload, or
65 3 Tom Clegg
* both of the above.
66 3 Tom Clegg
67 3 Tom Clegg
(to be discussed)
68 3 Tom Clegg
69 3 Tom Clegg
Each non-state-changing event should include the UUID of the relevant Log object.
70 3 Tom Clegg
71 3 Tom Clegg
Each non-state-changing event should include the attributes of the relevant Log object itself.
72 3 Tom Clegg
73 1 Tom Clegg
h1. Additional features
74 1 Tom Clegg
75 1 Tom Clegg
h2. Event sequence
76 1 Tom Clegg
77 1 Tom Clegg
With the current API server, it may be possible to update an object twice in quick succession such that the modification timestamps are out of order: i.e., the current state of object X has modification time T1, even though the same object previously had modification time T2>T1. If this occurs, the Events API must return the T2 update before the T1 update (or not return the T2 update at all).
78 1 Tom Clegg
79 1 Tom Clegg
In order to support delivery mechanisms where messages are re-ordered in transit, the Events API should assign a strictly increasing integer ID to each event sent over a given connection. Client pseudocode:
80 1 Tom Clegg
81 1 Tom Clegg
<pre>
82 1 Tom Clegg
receiveEvent(id, uuid, newEtag):
83 1 Tom Clegg
  if lastID[uuid] > id:
84 1 Tom Clegg
    # already received a newer update for this object
85 1 Tom Clegg
    return
86 1 Tom Clegg
  currentEtag[uuid] = newEtag
87 1 Tom Clegg
  lastID[uuid] = id
88 1 Tom Clegg
</pre>
89 1 Tom Clegg
90 1 Tom Clegg
Note these IDs are connection-specific: they cannot be used to reconnect and resume an event stream.
91 1 Tom Clegg
92 1 Tom Clegg
h2. Server-side event filters
93 1 Tom Clegg
94 1 Tom Clegg
Some clients will only be interested in a subset of possible changes. For example, a pipeline runner wants to know as soon as a container's "state" attribute changes, but might not care about other changes like "priority" or "progress".
95 1 Tom Clegg
96 1 Tom Clegg
Possible API features for reducing unnecessary work and network traffic:
97 1 Tom Clegg
# Allow clients to describe which attributes are interesting, e.g., @"select":["state"]@
98 1 Tom Clegg
# With each event, provide the list of changed attributes, e.g., @"changed":["state","output","log"]@, but not the attribute values themselves
99 1 Tom Clegg
100 1 Tom Clegg
These features might be tricky to implement efficiently for attributes that are computed on the fly. 
101 1 Tom Clegg
102 3 Tom Clegg
h2. Including object attributes with events
103 3 Tom Clegg
104 3 Tom Clegg
Some clients perform a GET request in response to every event reveived. For the sake of efficiency and convenience, if desired by the client, the Events API should perform that request internally, and supply the response along with the event.
105 3 Tom Clegg
106 3 Tom Clegg
Clients should be able to control (separately for each stream) the list of object attributes to include with each event. This list corresponds to the "select" parameter for the "get object" REST API.
107 3 Tom Clegg
108 3 Tom Clegg
By default, only the "uuid" and "etag" attributes are included. It is not possible to un-select those attributes.
109 3 Tom Clegg
110 3 Tom Clegg
The values for any returned attributes must be identical to the values that would be returned in a GET response.
111 3 Tom Clegg
112 1 Tom Clegg
h2. Null stream
113 1 Tom Clegg
114 1 Tom Clegg
To simplify implementation of clients that subscribe to event streams but also retrieve some objects without listening for events, a client should be able to use the Events API to retrieve the current state of an object without subscribing to the object's event stream.
115 2 Tom Clegg
116 2 Tom Clegg
h2. Ownership-change events
117 2 Tom Clegg
118 2 Tom Clegg
Some clients need to know when an object is added or removed from a project.
119 2 Tom Clegg
120 2 Tom Clegg
When an object's owner_uuid changes, this event should be sent to:
121 2 Tom Clegg
# all clients subscribed to the object itself
122 1 Tom Clegg
# all clients subscribed to the old owner_uuid
123 1 Tom Clegg
# all clients subscribed to the new owner_uuid
124 1 Tom Clegg
125 1 Tom Clegg
Likewise, subscribing to stream X should cause clients to receive messages when a new object is created with owner_uuid=X, and when an object with owner_uuid=X is deleted.
126 3 Tom Clegg
127 3 Tom Clegg
h2. Event batches
128 3 Tom Clegg
129 3 Tom Clegg
When sending a sequence of events that differ only in etag (i.e., they refer to the same object UUID and the payload consists of just the new etag) the server is permitted to send just the last event, and silently skip the rest.