Arvados

Regarding fetching the log from keep: How about the following?

• If the log collection is only of 1 block in size, fetch it and display it

• If the log collection has multiple blocks, fetch only the last block and display it

Branch 9514-delete-old-container-logs ready for review at 509f028

• Added a separate delete_old_container_logs task and tests

• Moved job related code from log_viewer.js into a separate job_log_viewer.js. Moved previous addToLogViewer code into job_log_viewer and renamed as addToJobLogViewer and converted addToLogViewer to handle generic message display

• Added log-viewer div and code similar to job log viewer into views/work_units/_show_log.html.erb

• Added integration tests (couldn’t accomplish it with a controller test due to javascript)

Reviewing 509f028

• (as mentioned in chat) seems like it would be simpler and more efficient to issue a single delete statement, instead of deleting in batches. I checked the log and this is producing statements like:

DELETE FROM "logs" WHERE "logs"."id" IN (139, 141, 142, ... (1000s of ids) ... )

• I'm not sure the log viewer loading the contents of crunch-run.txt is providing any value. Rather than creeping scope with a bunch of suggestions, maybe we should just disable it for crunch v2 logs until we can give it the attention it deserves. (We can keep the other refactoring.)

Branch 9514-only-delete-old-container-logs

"seems like it would be simpler and more efficient to issue a single delete statement ..." :

I was able to do this with pure SQL with using ActiveRecord syntax. Example SQL statement from API server log when the test runs:

 DELETE FROM logs WHERE id in (SELECT logs.id FROM logs JOIN containers ON logs.object_uuid = containers.uuid WHERE event_type IN ('stdout', 'stderr', 'arv-mount', 'crunch-run', 'crunchstat') AND containers.log IS NOT NULL AND containers.finished_at < '2015-09-30 23:34:14 UTC')

"I'm not sure the log viewer loading the contents of crunch-run.txt is providing any value ..." :

In this new branch, only deleting old logs. Updated the #Log for containers to not show the log div in terminal when it is empty

Peter: I was able to get the delete operation work with pure SQL, ditching the ActiveRecord syntax. Updated note 8 to reflect that. Thanks for prompting :).

Radhika Chippada wrote:

Peter: I was able to get the delete operation work with pure SQL, ditching the ActiveRecord syntax. Updated note 8 to reflect that. Thanks for prompting :).

Much better. LGTM @ f40544f6523bf2d54b288a64af7cab7469741512

At e1d958b

Updated the delete_old_job_logs task also to use this better performing SQL. The log statement from API server when tests are run:

DELETE FROM logs WHERE id in (SELECT logs.id FROM logs JOIN jobs ON logs.object_uuid = jobs.uuid WHERE event_type = 'stderr' AND jobs.log IS NOT NULL AND jobs.finished_at < '2015-10-02 14:14:04 UTC')