Project

General

Profile

Actions

Idea #16442

closed

Scalable + reliable container logging

Added by Peter Amstutz over 4 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
03/15/2023
Due date:
08/31/2023
Story points:
-
Release:
Release relationship:
Auto

Description

Objectives, proposed architecture, and implementation notes: Efficient live access to container logs


Related issues

Related to Arvados - Feature #18790: Access live container logs through arvados-client and crunch-run container gatewayResolvedTom Clegg04/04/2023Actions
Related to Arvados - Feature #19889: access current container logs at /arvados/v1/containers/{uuid}/log/{filename}ResolvedTom Clegg03/23/2023Actions
Related to Arvados - Feature #20217: crunch-run WebDAV service providing access to latest logsDuplicateActions
Related to Arvados - Feature #20218: Container log endpoint in controller that routes to either crunch-run or keep-webDuplicateActions
Related to Arvados - Feature #20219: Log panel on container view fetches live logs using periodic range request on new container log endpointResolvedStephen Smith07/28/2023Actions
Related to Arvados - Feature #20220: Dispatcher uses live logs endpoint on crunch-run to fetch logs and store a backup locallyNewActions
Related to Arvados - Idea #20258: "Live" log view should show users the most recent logs, wherever those areResolved03/06/202303/06/2023Actions
Related to Arvados - Feature #20310: Add server-sent events stream at .../containers/{uuid}/log_eventsNewActions
Related to Arvados - Feature #20319: Move containers/{uuid}/log/ to container_requests/{uuid}/log/ResolvedTom Clegg04/21/2023Actions
Related to Arvados - Feature #20433: Config option to have crunch-run stop sending logs to logs tableResolvedBrett Smith05/31/2023Actions
Related to Arvados - Idea #20458: Document arvados-client logs commandNewActions
Related to Arvados - Bug #20804: crunchstat-summary should use container logs API, not CollectionReader and logs tableNewActions
Related to Arvados - Idea #20955: Crunch log transition documentation issuesResolvedPeter Amstutz09/15/2023Actions
Has duplicate Arvados - Idea #16222: Handle container live logs in a more scalable wayResolvedActions
Blocks Arvados Epics - Idea #20599: Scaling to 1000s of concurrent containersResolved06/01/202303/31/2024Actions
Actions #1

Updated by Peter Amstutz over 4 years ago

  • Related to Idea #16222: Handle container live logs in a more scalable way added
Actions #2

Updated by Peter Amstutz over 4 years ago

  • Start date set to 06/01/2021
  • Due date set to 09/01/2021
Actions #3

Updated by Peter Amstutz almost 4 years ago

  • Start date changed from 06/01/2021 to 01/01/2022
  • Due date changed from 09/01/2021 to 03/31/2022
Actions #4

Updated by Peter Amstutz about 3 years ago

  • Start date changed from 01/01/2022 to 06/01/2022
  • Due date changed from 03/31/2022 to 09/30/2022
Actions #5

Updated by Peter Amstutz almost 3 years ago

  • Start date changed from 06/01/2022 to 09/01/2022
  • Due date changed from 09/30/2022 to 12/31/2022
Actions #6

Updated by Tom Clegg over 2 years ago

  • Related to Feature #18790: Access live container logs through arvados-client and crunch-run container gateway added
Actions #7

Updated by Tom Clegg over 2 years ago

  • Description updated (diff)
  • Subject changed from Redesign live container logs to Scalable + reliable container logging
Actions #8

Updated by Peter Amstutz over 2 years ago

  • Start date changed from 09/01/2022 to 12/01/2022
  • Due date changed from 12/31/2022 to 03/31/2023
Actions #9

Updated by Tom Clegg almost 2 years ago

  • Description updated (diff)
Actions #10

Updated by Tom Clegg almost 2 years ago

  • Related to Feature #19889: access current container logs at /arvados/v1/containers/{uuid}/log/{filename} added
Actions #11

Updated by Peter Amstutz almost 2 years ago

  • Start date changed from 12/01/2022 to 06/01/2023
  • Due date changed from 03/31/2023 to 12/31/2023
Actions #12

Updated by Peter Amstutz over 1 year ago

  • Related to Feature #20217: crunch-run WebDAV service providing access to latest logs added
Actions #13

Updated by Peter Amstutz over 1 year ago

  • Related to Feature #20218: Container log endpoint in controller that routes to either crunch-run or keep-web added
Actions #14

Updated by Peter Amstutz over 1 year ago

  • Related to Feature #20219: Log panel on container view fetches live logs using periodic range request on new container log endpoint added
Actions #15

Updated by Peter Amstutz over 1 year ago

  • Related to Feature #20220: Dispatcher uses live logs endpoint on crunch-run to fetch logs and store a backup locally added
Actions #16

Updated by Peter Amstutz over 1 year ago

  • Start date changed from 06/01/2023 to 04/01/2023
  • Due date changed from 12/31/2023 to 07/31/2023
Actions #17

Updated by Brett Smith over 1 year ago

  • Related to Idea #20258: "Live" log view should show users the most recent logs, wherever those are added
Actions #18

Updated by Peter Amstutz over 1 year ago

  • Start date changed from 04/01/2023 to 03/15/2023
Actions #19

Updated by Peter Amstutz over 1 year ago

  • Due date changed from 07/31/2023 to 05/31/2023
Actions #20

Updated by Tom Clegg over 1 year ago

  • Related to Feature #20310: Add server-sent events stream at .../containers/{uuid}/log_events added
Actions #21

Updated by Tom Clegg over 1 year ago

  • Related to Feature #20319: Move containers/{uuid}/log/ to container_requests/{uuid}/log/ added
Actions #22

Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress
Actions #23

Updated by Tom Clegg over 1 year ago

  • Related to Feature #20433: Config option to have crunch-run stop sending logs to logs table added
Actions #24

Updated by Tom Clegg over 1 year ago

  • Related to Idea #20458: Document arvados-client logs command added
Actions #25

Updated by Tom Clegg over 1 year ago

  • Related to deleted (Idea #16222: Handle container live logs in a more scalable way)
Actions #26

Updated by Tom Clegg over 1 year ago

  • Has duplicate Idea #16222: Handle container live logs in a more scalable way added
Actions #27

Updated by Peter Amstutz over 1 year ago

  • Blocks Idea #20599: Scaling to 1000s of concurrent containers added
Actions #28

Updated by Peter Amstutz over 1 year ago

  • Due date changed from 05/31/2023 to 06/30/2023
Actions #31

Updated by Peter Amstutz over 1 year ago

  • Due date changed from 06/30/2023 to 07/31/2023
Actions #32

Updated by Tom Clegg over 1 year ago

  • Related to Bug #20804: crunchstat-summary should use container logs API, not CollectionReader and logs table added
Actions #33

Updated by Peter Amstutz about 1 year ago

  • Due date changed from 07/31/2023 to 08/31/2023
Actions #34

Updated by Brett Smith about 1 year ago

  • Related to Idea #20955: Crunch log transition documentation issues added
Actions #35

Updated by Peter Amstutz about 1 year ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF