Project

General

Profile

Actions

Feature #20220

open

Dispatcher uses live logs endpoint on crunch-run to fetch logs and store a backup locally

Added by Peter Amstutz 9 months ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Crunch
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

https://dev.arvados.org/projects/arvados/wiki/Efficient_live_access_to_container_logs

Dispatcher should periodically contact each running container and fetch the sizes of the live logs. For all the logs that have new data, fetch the data and write them to files local to the dispatcher.

When the container is about to go into a cancelled or complete state, compare the container log collection to the local files. If the local files have content that isn't in the log collection, update the log collection. Then local logs can be discarded.


Related issues

Related to Arvados Epics - Story #16442: Scalable + reliable container loggingResolved03/15/202308/31/2023

Actions
Actions #1

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz 9 months ago

  • Related to Story #16442: Scalable + reliable container logging added
Actions

Also available in: Atom PDF