Feature #18863

[controller] add background job to clean up old container log records

Added by Ward Vandewege 2 months ago. Updated 19 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

As identified in https://dev.arvados.org/issues/18763#note-5, we have a "deleted_old_container_logs" rake task that is supposed to be running in a cron job to clear out old container logs.

Following the pattern we started using for the trash sweeps (#18339), add a background job that executes this sql query.

Remove the rake task file from the repository. Add a note to the upgrade nodes document that the cron job should be removed when upgrading.

The query used by the existing rake task is

DELETE FROM logs WHERE id in (SELECT logs.id FROM logs JOIN containers ON logs.object_uuid = containers.uuid WHERE event_type IN ('stdout', 'stderr', 'arv-mount', 'crunch-run', 'crunchstat') AND containers.log IS NOT NULL AND now() - containers.finished_at > interval '#{Rails.configuration.Containers.Logging.MaxAge.to_i} seconds')"

Determined empirically, a far more efficient version of that (if postgresql specific) is

delete from logs using containers where logs.object_uuid=containers.uuid and logs.event_type in ('stdout', 'stderr', 'arv-mount', 'crunch-run', 'crunchstat') AND containers.log IS NOT NULL AND now() - containers.finished_at > interval '#{Rails.configuration.Containers.Logging.MaxAge.to_i} seconds')"

Don't forget to remove the delete_old_container_logs.rake file from the API server after the new background job is implemented, cf. https://dev.arvados.org/issues/18763#note-5


Related issues

Related to Arvados - Bug #18763: remove unused rake tasksResolved03/14/2022

Related to Arvados - Bug #18762: rails background tasks scaling issuesNew

History

#1 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#2 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#3 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#4 Updated by Ward Vandewege 2 months ago

  • Related to Bug #18763: remove unused rake tasks added

#5 Updated by Peter Amstutz 2 months ago

  • Target version changed from 2022-03-30 Sprint to 2022-04-13 Sprint

#6 Updated by Peter Amstutz 2 months ago

  • Related to Bug #18762: rails background tasks scaling issues added

#7 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2022-04-13 Sprint to 2022-04-27 Sprint

#8 Updated by Peter Amstutz about 1 month ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint

#9 Updated by Peter Amstutz 29 days ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint

#10 Updated by Peter Amstutz 19 days ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint

#11 Updated by Peter Amstutz 19 days ago

  • Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint

#12 Updated by Peter Amstutz 19 days ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06

Also available in: Atom PDF