Arvados

• Teach Workbench to watch for state changes, and display them as text in the logging area
• Explicitly log a chunk of text from crunch-dispatch-* and crunch-run when something interesting happens (e.g., event_type="dispatch", properties={"text":"locked by dispatcher pid=%d"})

In the current setup we write HTML for the "terminal window" widget in two different places: once in Ruby where we first render the log tab, and again in JavaScript when we append new logs as they arrive via websocket. So far, this seems reasonable because the JS side consists of appending text to a "pre" element. However, if we want to do something beyond just copying text from the log entries -- e.g., notice event_type=update logs and look in old/new_attributes to see whether state changed -- keeping the Ruby and JS implementations synchronized might get annoying fast.

1. Do this on the client side (in JS), so these logs will be displayed if they arrive while you have the page open, but if you hit Refresh (or arrive at the page too late) you'll miss them.
2. Port/copy the code to Ruby as well. Add comments and tests to assist with "change X whenever you change Y".
3. Don't pre-render the log content in the Ruby code. Instead, use client-side code to retrieve older log events from the API server.

I think the best outcome is (1)+(3), but if it's going to be a while before we get around to (3), we should probably do (2) in the meantime.

9679-dispatch-event-logs @ b046d1f

...logs a message in Workbench when a container changes state, e.g.,

2016-09-06T14:10:31.506744197Z Container zzzzz-dz642-queuedcontainer was returned to the queue
2016-09-06T14:10:31.516115221Z Container zzzzz-dz642-queuedcontainer was taken from the queue by a dispatch process
2016-09-06T14:10:31.536427858Z Container zzzzz-dz642-queuedcontainer started
2016-09-06T14:10:31.546526907Z Container zzzzz-dz642-queuedcontainer finished with exit code 1 (failure)

Until we do (2) or (3) in note-7 these messages appear only if the log tab is open when those events occur. This isn't ideal, but it does address the problem of the user seeing nothing at all when there's a problem running slurm jobs.

• Just one question. In work_unit_log.js => var stamp = eventData.event_at + " " -- Are we guaranteed that any “properties.text” is terminated with a new line? That is, do we need to prepend the eventData.event_at with a “ “ when there is text in properties?

• I didn't run any of the tests and am assuming you tested already :)

LGTM

The previous code assumed log.properties.text always contains all relevant newlines, but now that we're inserting other stuff between logs, it makes more sense to enforce a line break after each entry. Added a trailing newline when needed.