Project

General

Profile

Actions

Feature #18863

closed

add background job to controller to clean up old log records

Added by Ward Vandewege 11 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
10/18/2022
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

As identified in https://dev.arvados.org/issues/18763#note-5, we have a "deleted_old_container_logs" rake task that is supposed to be running in a cron job to clear out old container logs.

Following the pattern we started using for the trash sweeps (#18339), add a background job that executes this sql query.

Remove the rake task file from the repository. Add a note to the upgrade nodes document that the cron job should be removed when upgrading.

The query used by the existing rake task is

DELETE FROM logs WHERE id in (SELECT logs.id FROM logs JOIN containers ON logs.object_uuid = containers.uuid WHERE event_type IN ('stdout', 'stderr', 'arv-mount', 'crunch-run', 'crunchstat') AND containers.log IS NOT NULL AND now() - containers.finished_at > interval '#{Rails.configuration.Containers.Logging.MaxAge.to_i} seconds')"

Determined empirically, a far more efficient version of that (if postgresql specific) is

delete from logs using containers where logs.object_uuid=containers.uuid and logs.event_type in ('stdout', 'stderr', 'arv-mount', 'crunch-run', 'crunchstat') AND containers.log IS NOT NULL AND now() - containers.finished_at > interval '#{Rails.configuration.Containers.Logging.MaxAge.to_i} seconds')"

Don't forget to change the delete_old_container_logs.rake file from the API server to a no-op after the new background job is implemented, https://dev.arvados.org/issues/18763#note-5

There are separate lifetimes for container logs and general audit logs, ensure that both are honored.


Subtasks 1 (0 open1 closed)

Task #19608: Review 18863-delete-old-logs-without-cronResolvedTom Clegg10/18/2022

Actions

Related issues

Related to Arvados - Bug #18763: remove unused rake tasksResolvedWard Vandewege03/14/2022

Actions
Related to Arvados - Bug #18762: rails background tasks scaling issuesNew

Actions
Actions #1

Updated by Ward Vandewege 11 months ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege 11 months ago

  • Description updated (diff)
Actions #3

Updated by Ward Vandewege 11 months ago

  • Description updated (diff)
Actions #4

Updated by Ward Vandewege 11 months ago

  • Related to Bug #18763: remove unused rake tasks added
Actions #5

Updated by Peter Amstutz 11 months ago

  • Target version changed from 2022-03-30 Sprint to 2022-04-13 Sprint
Actions #6

Updated by Peter Amstutz 11 months ago

  • Related to Bug #18762: rails background tasks scaling issues added
Actions #7

Updated by Peter Amstutz 11 months ago

  • Target version changed from 2022-04-13 Sprint to 2022-04-27 Sprint
Actions #8

Updated by Peter Amstutz 10 months ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Actions #9

Updated by Peter Amstutz 10 months ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Actions #10

Updated by Peter Amstutz 9 months ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Actions #11

Updated by Peter Amstutz 9 months ago

  • Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint
Actions #12

Updated by Peter Amstutz 9 months ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06
Actions #13

Updated by Peter Amstutz 8 months ago

  • Target version changed from 2022-07-06 to 2022-07-20
Actions #14

Updated by Peter Amstutz 7 months ago

  • Target version changed from 2022-07-20 to 2022-08-03 Sprint
Actions #15

Updated by Peter Amstutz 7 months ago

  • Target version changed from 2022-08-03 Sprint to 2022-08-17 sprint
Actions #16

Updated by Peter Amstutz 6 months ago

  • Target version changed from 2022-08-17 sprint to 2022-08-31 sprint
Actions #17

Updated by Peter Amstutz 6 months ago

  • Target version changed from 2022-08-31 sprint to 2022-09-14 sprint
Actions #18

Updated by Peter Amstutz 5 months ago

  • Target version changed from 2022-09-14 sprint to 2022-09-28 sprint
Actions #19

Updated by Peter Amstutz 5 months ago

  • Target version changed from 2022-09-28 sprint to 2022-10-12 sprint
Actions #20

Updated by Peter Amstutz 4 months ago

  • Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Actions #21

Updated by Peter Amstutz 4 months ago

  • Target version changed from 2022-10-26 sprint to 2022-10-12 sprint
Actions #22

Updated by Peter Amstutz 4 months ago

  • Assigned To set to Tom Clegg
Actions #23

Updated by Peter Amstutz 4 months ago

  • Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Actions #24

Updated by Peter Amstutz 4 months ago

  • Category set to API
  • Subject changed from [controller] add background job to clean up old container log records to add background job to controller to clean up old container log records
Actions #25

Updated by Peter Amstutz 4 months ago

  • Description updated (diff)
  • Subject changed from add background job to controller to clean up old container log records to add background job to controller to clean up old log records
Actions #26

Updated by Tom Clegg 4 months ago

18863-delete-old-logs-without-cron @ b89ab7f9270acfabe9139d14d7071cf117b39bd4 -- developer-run-tests: #3324

  • configurable "delete old logs" interval
  • controller background task
  • rake task is now a no-op (but still exists, so any existing cron jobs succeed harmlessly instead of failing and causing unnecessary sysadmin alerts)

Currently, old audit logs (e.g., "collection was updated") are automatically deleted by a background task in railsapi that's kicked off by an after_commit hook. Eventually we want to move that to Go as well, but I'm assuming that's out of scope here, because moving it wouldn't improve/simplify arvados installation.

I can't find any mention in the existing install docs about running the rake task as a cron job. Did we just neglect to document this at all?

Actions #27

Updated by Tom Clegg 4 months ago

  • Status changed from New to In Progress
Actions #28

Updated by Lucas Di Pentima 4 months ago

Just one comment:

Only 5 different event_type logs get cleaned up in this branch. While working on WB2's log viewer I noticed there're some additional ones, do you think the ones listed here also need sweeping? https://dev.arvados.org/projects/arvados/repository/arvados-workbench2/revisions/main/entry/src/store/process-logs-panel/process-logs-panel-actions.ts#L134

The rest LGTM, thanks.

Actions #29

Updated by Tom Clegg 4 months ago

Good catch. Added the other event_type values.

18863-delete-old-logs-without-cron @ f3203e42412d5b5216b2c70caae47b73b712d18c -- developer-run-tests: #3327

Actions #30

Updated by Tom Clegg 4 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #31

Updated by Peter Amstutz about 2 months ago

  • Release set to 47
Actions

Also available in: Atom PDF