Project

General

Profile

Actions
Copy link

Support #16417

closed

Salt installer includes setting up log aggregation with loki

Added by Peter Amstutz over 4 years ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Lucas Di Pentima
Category:
Deployment
Target version:
Development 2024-11-06 sprint
Due date:
Story points:
-
Release:
Arvados 3.0
Release relationship:
Auto

Description

Centralize collection of service logs on a cluster so they are easier to access and available if the system crashes.

Subtasks 2 (0 open — 2 closed)

Task #22123: Review 16417-installer-lokiResolvedLucas Di Pentima10/10/2024Actions
Task #22237: Review follow-up branch 16417-installer-loki-fixesResolvedLucas Di Pentima10/23/2024Actions
Actions
Copy link
 #1

Updated by Peter Amstutz over 4 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Peter Amstutz over 4 years ago

  • Tracker changed from Bug to Support
Actions
Copy link
 #3

Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint
Actions
Copy link
 #4

Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint
Actions
Copy link
 #5

Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2020-07-01 Sprint to 2020-07-15
Actions
Copy link
 #6

Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2020-07-15 to 2020-08-26 Sprint
Actions
Copy link
 #7

Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint
Actions
Copy link
 #8

Updated by Javier Bértoli over 4 years ago

  • Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint
Actions
Copy link
 #9

Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2020-09-23 Sprint to 2020-10-21 Sprint
Actions
Copy link
 #10

Updated by Peter Amstutz about 4 years ago

  • Target version changed from 2020-10-21 Sprint to 2020-11-04 Sprint
Actions
Copy link
 #11

Updated by Peter Amstutz about 4 years ago

  • Target version deleted (2020-11-04 Sprint)
Actions
Copy link
 #12

Updated by Peter Amstutz almost 2 years ago

  • Release set to 60
Actions
Copy link
 #13

Updated by Peter Amstutz 10 months ago

  • Target version set to Future
Actions
Copy link
 #14

Updated by Peter Amstutz 4 months ago

  • Release deleted (60)
  • Target version changed from Future to Development 2024-09-25 sprint
  • Assigned To changed from Javier Bértoli to Lucas Di Pentima
  • Subject changed from Centralize logs to Salt installer includes setting up log aggregation with loki
Actions
Copy link
 #15

Updated by Peter Amstutz 4 months ago

  • Category set to Deployment
  • Project changed from 35 to Arvados
Actions
Copy link
 #16

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-09-25 sprint to Development 2024-10-09 sprint
Actions
Copy link
 #17

Updated by Lucas Di Pentima 3 months ago

  • Status changed from New to In Progress
Actions
Copy link
 #18

Updated by Lucas Di Pentima 3 months ago

16417-installer-loki @ 84ba7b0687

test-provision: #980

  • All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
    • Yes
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • No
  • Code is tested and passing, both automated and manual, what manual testing was done is described.
    • Manual test performed by deploying a multi node cluster and confirming that everything got installed properly
  • New or changed UX/UX and has gotten feedback from stakeholders.
    • N/A
  • Documentation has been updated.
    • Yes
  • Behaves appropriately at the intended scale (describe intended scale).
    • No change in scale
  • Considered backwards and forwards compatibility issues between client and server.
    • N/A
  • Follows our coding standards and GUI style guidelines.
    • N/A
  • Terraform changes include:
    • Adding a new DNS name for the loki URL
    • Creating a new S3 bucket to hold Loki data and access resources for it.
  • Installer changes include:
    • Adding new nginx configurations for the loki URL
    • Installing and configuring the Loki service in the monitoring role
    • Installing and configuring the Grafana Alloy service on all nodes
    • Installing the Loki related dashboard on Grafana
Actions
Copy link
 #19

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-10-09 sprint to Development 2024-10-23 sprint
Actions
Copy link
 #20

Updated by Brett Smith 3 months ago

Lucas Di Pentima wrote in #note-18:

16417-installer-loki @ 84ba7b0687

LGTM, thanks.

Actions
Copy link
 #21

Updated by Lucas Di Pentima 3 months ago

  • Status changed from In Progress to Resolved
Actions
Copy link
 #22

Updated by Lucas Di Pentima 3 months ago

  • Status changed from Resolved to In Progress

Re-opening this as I've discovered a bug in the installer code that makes grafana installed on all nodes.

Actions
Copy link
 #23

Updated by Lucas Di Pentima 2 months ago

Updates at c3e3074 - 16417-installer-loki-fixes branch
Test run: test-provision: #1012

  • Instead of requiring grafana-formula's managed package repo state, alloy_install state now manages it directly. This avoids having grafana installed on all nodes.
  • While manually testing a multi-node cluster deployment, Loki was failing to start because the auto-generated S3 bucket password included 2 / chars that cannot parse correctly in its config file. Now we urlencode() the password to fix it.
Actions
Copy link
 #24

Updated by Peter Amstutz 2 months ago

  • Release set to 70
Actions
Copy link
 #25

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-10-23 sprint to Development 2024-11-06 sprint
Actions
Copy link
 #26

Updated by Brett Smith 2 months ago

Lucas Di Pentima wrote in #note-23:

Updates at c3e3074 - 16417-installer-loki-fixes branch
Test run: test-provision: #1012

LGTM, thanks.

Actions
Copy link
 #27

Updated by Lucas Di Pentima 2 months ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF