Added the following on top of previously implemented and reviewed set: - Update crunch-dispatch.rb to send redis logs to Logs table. Used event type "transient-log-entry" to indicate that these are transient rows and can be removed after the job(s) complete - Updated the event_log.js to also support data-object-uuid*s*. - Update pipeline instance to display jobs related log entries using data-object-uuid*s* - Added helper method to pipeline_instance_helper to get the log history. - In pi log display, display the history (from above) before appending real time logs - Display log links when the pi is in "done" state

Features * Display folders as a hierarchy at /folders (instead of a flat list). * Add a "move to..." button to the "show folder" page so you can control the structure of the folder hierarchy.

This 2678 branch is off of the 2756-eventbus-in-workbench so that the crunch-dispatch updates from that branch are carried over. Please compare against the 2756-eventbus-in-workbench branch when you do a diff. 1. a-r-p-i updated to set the group's owner_uuid on the newly created job 2. crunch-dispatch is updated to set the group's owner_uuid on the Log lines Question: The log.txt and output are created (arv keep put), and their links are created from the crunch-job script. It appears that we need to update that script to set the owner_uuid on the log.txt and output files. Is this correct?

Changes @ commit:5cdf402cf Successful GET or HEAD responses include an @X-Block-Size@ header.

http://doc.arvados.org/user/tutorials/tutorial-firstscript.html ~$ git clone git@git.qr1hi.arvadosapi.com:jsheffi.git Cloning into 'jsheffi'... The authenticity of host 'git.qr1hi.arvadosapi.com (10.13.26.53)' can't be established. RSA key fingerprint is a6:b0:be:4d:7a:5d:4e:ba:b4:04:17:82:9f:ee:40:dc. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'git.qr1hi.arvadosapi.com,10.13.26.53' (RSA) to the list of known hosts. warning: You appear to have cloned an empty repository. I would make sure to note that this warning will come up, is expected behavior, and that the user should say yes, and that the following warnings are ok to get.

http://doc.arvados.org/user/tutorials/tutorial-firstscript.html ~$ cd you ~/you$ mkdir crunch_scripts ~/you$ cd crunch_scripts Need to be told: a) to start in the home directory, and b) to "mkdir you"

http://doc.arvados.org/user/tutorials/tutorial-firstscript.html jsheffi@shell.qr1hi:~/jsheffi/crunch_scripts$ nano hash.py jsheffi@shell.qr1hi:~/jsheffi/crunch_scripts$ chmod +x hash.py jsheffi@shell.qr1hi:~/jsheffi/crunch_scripts$ git add hash.py fatal: Not a git repository (or any of the parent directories): .git

http://doc.arvados.org/user/tutorials/tutorial-firstscript.html The very beginning of this page needs to say "cd" so that you go back from scratch/you to ~. Otherwise everything after this part can end up being wrong.

http://doc.arvados.org/user/tutorials/tutorial-firstscript.html tutorial says: ~/you/crunch_scripts$ git commit -m"my first script" [master (root-commit) 27fd88b] my first script 1 file changed, 33 insertions(+) create mode 100755 crunch_scripts/hash.py my terminal says: jsheffi@shell.qr1hi:~/jsheffi/crunch_scripts$ git commit -m"my first script" [master (root-commit) 8032ec3] my first script 1 file changed, 45 insertions(+) create mode 100755 crunch_scripts/hash.py

http://doc.arvados.org/user/topics/running-pipeline-command-line.html jsheffi@shell.qr1hi:~$ arv keep get 8e1b6acdd3f2f1da722538127c5c6202+56/md5sum.txt WARNING:root:API lookup failed for collection 8e1b6acdd3f2f1da722538127c5c6202+56 (<class 'apiclient.errors.HttpError'>: <HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/8e1b6acdd3f2f1da722538127c5c6202%2B56?alt=json returned "Not Found">) 0f1d6bcf55c34bed7f92a805d2d89bbf alice.txt 504938460ef369cd275e4ef58994cffe bob.txt 8f3b36aff310e06f3c5b9e95678ff77a carol.txt

The documentation says "Request a new VM" without explanation http://doc.arvados.org/user/index.html Later it says "log into a VM" without explaning how to determine the hostname of the appropriate VM to use. http://doc.arvados.org/user/tutorial-job1.html

http://doc.arvados.org/user/tutorials/tutorial-keep.html command: arv keep get 887cd41e9c613463eab2f0d885c6dd96+83/alice.txt . result: WARNING:root:API lookup failed for collection 887cd41e9c613463eab2f0d885c6dd96+83 (<class 'apiclient.errors.HttpError'>: <HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/887cd41e9c613463eab2f0d885c6dd96%2B83?alt=json returned "Not Found">) Also, the word "indvidual" is misspelled.

Just a couple very minor doc text updates. In addition, added links in prod to share the collections mentioned in the tutorial sections.

Present a special index at @https://WORKBENCH/collections/download/:uuid/:reader_token/*file@. This index has links to all the files, and nothing else. The goal here is that, when you point @wget -r@ at this URL, you get pretty results on disk.

SIGTERM support for Keep.

Tasks that are helpful for debugging or deploying Keep to staging. Features include: * Keep should terminate gracefully on SIGTERM. * Improved logging. Events Keep should log: * Requests. Each request should generate one line to a request log a la apache logging. * Errors. Errors and warnings should be sent to an error log. * Debugging. Debugging messages should be generated at a level, e.g. @debug.Log(level_num, msg)@ The level should be configurable via -v -v -v command-line options or possibly -v=3 The administrator should be able to configure the destination of these log messages, or turn them off entirely.

Being able to dynamically open and close folders is nice but not required.

This branch adds relations information to Workbench's Collection view, as described in task #2780.

Show additional information from the API server that immediately pertains to this collection: name, folders that contain this, where this came from (e.g., Job output, Job log)

Add temporary support for old clients by providing computed active/success flags in API responses, and changing to the appropriate state if an update request has resource_attrs[:success]==true, etc.

Tom, Please review the "second" phase, where the arv-run-pipeline-instance is updated to handle RunningOnClient and Paused states. A couple questions: 1. In pipeline_instance.rb: how about I merge the code from the "verify_status" method into "update_success"? I can call this method as "update_success_and_verify_status". Concern being, I am not sure if it is guaranteed that update_success method is executed "before" the verify_status method and if there is a situation where this becomes an issue otherwise. 2. components_summary: # 1. Do we want to call "done" as "succeeded"? # 2. How to determine "running"?

this branch handles phase 3 of the story, where workbench and arv-run-pipeline-instance scripts use pipeline_instance -> state and not use active and success flags.