Project

General

Profile

Actions

Feature #18863

closed

add background job to controller to clean up old log records

Added by Ward Vandewege about 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Story points:
-
Release relationship:
Auto

Description

As identified in https://dev.arvados.org/issues/18763#note-5, we have a "deleted_old_container_logs" rake task that is supposed to be running in a cron job to clear out old container logs.

Following the pattern we started using for the trash sweeps (#18339), add a background job that executes this sql query.

Remove the rake task file from the repository. Add a note to the upgrade nodes document that the cron job should be removed when upgrading.

The query used by the existing rake task is

DELETE FROM logs WHERE id in (SELECT logs.id FROM logs JOIN containers ON logs.object_uuid = containers.uuid WHERE event_type IN ('stdout', 'stderr', 'arv-mount', 'crunch-run', 'crunchstat') AND containers.log IS NOT NULL AND now() - containers.finished_at > interval '#{Rails.configuration.Containers.Logging.MaxAge.to_i} seconds')"

Determined empirically, a far more efficient version of that (if postgresql specific) is

delete from logs using containers where logs.object_uuid=containers.uuid and logs.event_type in ('stdout', 'stderr', 'arv-mount', 'crunch-run', 'crunchstat') AND containers.log IS NOT NULL AND now() - containers.finished_at > interval '#{Rails.configuration.Containers.Logging.MaxAge.to_i} seconds')"

Don't forget to change the delete_old_container_logs.rake file from the API server to a no-op after the new background job is implemented, https://dev.arvados.org/issues/18763#note-5

There are separate lifetimes for container logs and general audit logs, ensure that both are honored.


Subtasks 1 (0 open1 closed)

Task #19608: Review 18863-delete-old-logs-without-cronResolvedTom Clegg10/18/2022Actions

Related issues

Related to Arvados - Bug #18763: remove unused rake tasksResolvedWard Vandewege03/14/2022Actions
Related to Arvados - Bug #18762: rails background tasks scaling issuesNewActions
Actions #1

Updated by Ward Vandewege about 2 years ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege about 2 years ago

  • Description updated (diff)
Actions #3

Updated by Ward Vandewege about 2 years ago

  • Description updated (diff)
Actions #4

Updated by Ward Vandewege about 2 years ago

  • Related to Bug #18763: remove unused rake tasks added
Actions #5

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-03-30 Sprint to 2022-04-13 Sprint
Actions #6

Updated by Peter Amstutz about 2 years ago

  • Related to Bug #18762: rails background tasks scaling issues added
Actions #7

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-04-13 Sprint to 2022-04-27 Sprint
Actions #8

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Actions #9

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Actions #10

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Actions #11

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint
Actions #12

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06
Actions #13

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-07-06 to 2022-07-20
Actions #14

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-07-20 to 2022-08-03 Sprint
Actions #15

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-08-03 Sprint to 2022-08-17 sprint
Actions #16

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-08-17 sprint to 2022-08-31 sprint
Actions #17

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-08-31 sprint to 2022-09-14 sprint
Actions #18

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-09-14 sprint to 2022-09-28 sprint
Actions #19

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-09-28 sprint to 2022-10-12 sprint
Actions #20

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Actions #21

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-10-26 sprint to 2022-10-12 sprint
Actions #22

Updated by Peter Amstutz over 1 year ago

  • Assigned To set to Tom Clegg
Actions #23

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Actions #24

Updated by Peter Amstutz over 1 year ago

  • Category set to API
  • Subject changed from [controller] add background job to clean up old container log records to add background job to controller to clean up old container log records
Actions #25

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
  • Subject changed from add background job to controller to clean up old container log records to add background job to controller to clean up old log records
Actions #26

Updated by Tom Clegg over 1 year ago

18863-delete-old-logs-without-cron @ b89ab7f9270acfabe9139d14d7071cf117b39bd4 -- developer-run-tests: #3324

  • configurable "delete old logs" interval
  • controller background task
  • rake task is now a no-op (but still exists, so any existing cron jobs succeed harmlessly instead of failing and causing unnecessary sysadmin alerts)

Currently, old audit logs (e.g., "collection was updated") are automatically deleted by a background task in railsapi that's kicked off by an after_commit hook. Eventually we want to move that to Go as well, but I'm assuming that's out of scope here, because moving it wouldn't improve/simplify arvados installation.

I can't find any mention in the existing install docs about running the rake task as a cron job. Did we just neglect to document this at all?

Actions #27

Updated by Tom Clegg over 1 year ago

  • Status changed from New to In Progress
Actions #28

Updated by Lucas Di Pentima over 1 year ago

Just one comment:

Only 5 different event_type logs get cleaned up in this branch. While working on WB2's log viewer I noticed there're some additional ones, do you think the ones listed here also need sweeping? https://dev.arvados.org/projects/arvados/repository/arvados-workbench2/revisions/main/entry/src/store/process-logs-panel/process-logs-panel-actions.ts#L134

The rest LGTM, thanks.

Actions #29

Updated by Tom Clegg over 1 year ago

Good catch. Added the other event_type values.

18863-delete-old-logs-without-cron @ f3203e42412d5b5216b2c70caae47b73b712d18c -- developer-run-tests: #3327

Actions #30

Updated by Tom Clegg over 1 year ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #31

Updated by Peter Amstutz over 1 year ago

  • Release set to 47
Actions

Also available in: Atom PDF