Support #16417: Salt installer includes setting up log aggregation with loki - Arvados

Custom queries

All assigned issues
All issues assigned for grooming
My issues for grooming
My issues for grooming (no story pts)
Prioritized open issues

Actions

Copy link

Support #16417

closed

Salt installer includes setting up log aggregation with loki

Added by Peter Amstutz over 4 years ago. Updated 28 days ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Lucas Di Pentima

Category:

Deployment

Target version:

Development 2024-11-06 sprint

Due date:

Story points:

Release:

Arvados 3.0

Release relationship:

Auto

Description

Centralize collection of service logs on a cluster so they are easier to access and available if the system crashes.

Subtasks 2 (0 open — 2 closed)

	Task #22123: Review 16417-installer-loki	Resolved	Lucas Di Pentima	10/10/2024			Actions
	Task #22237: Review follow-up branch 16417-installer-loki-fixes	Resolved	Lucas Di Pentima	10/23/2024			Actions

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Tracker changed from Bug to Support

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Target version changed from 2020-07-01 Sprint to 2020-07-15

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Target version changed from 2020-07-15 to 2020-08-26 Sprint

Actions

Copy link

Updated by Peter Amstutz over 4 years ago

Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint

Actions

Copy link

Updated by Javier Bértoli about 4 years ago

Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint

Actions

Copy link

Updated by Peter Amstutz about 4 years ago

Target version changed from 2020-09-23 Sprint to 2020-10-21 Sprint

Actions

Copy link

#10

Updated by Peter Amstutz about 4 years ago

Target version changed from 2020-10-21 Sprint to 2020-11-04 Sprint

Actions

Copy link

#11

Updated by Peter Amstutz about 4 years ago

Target version deleted (~~2020-11-04 Sprint~~)

Actions

Copy link

#12

Updated by Peter Amstutz almost 2 years ago

Release set to 60

Actions

Copy link

#13

Updated by Peter Amstutz 9 months ago

Target version set to Future

Actions

Copy link

#14

Updated by Peter Amstutz 2 months ago

Release deleted (60)
Target version changed from Future to Development 2024-09-25 sprint
Assigned To changed from Javier Bértoli to Lucas Di Pentima
Subject changed from Centralize logs to Salt installer includes setting up log aggregation with loki

Actions

Copy link

#15

Updated by Peter Amstutz 2 months ago

Category set to Deployment
Project changed from 35 to Arvados

Actions

Copy link

#16

Updated by Peter Amstutz about 2 months ago

Target version changed from Development 2024-09-25 sprint to Development 2024-10-09 sprint

Actions

Copy link

#17

Updated by Lucas Di Pentima about 2 months ago

Status changed from New to In Progress

Actions

Copy link

#18

Updated by Lucas Di Pentima about 1 month ago

16417-installer-loki @ 84ba7b0687

test-provision: #980

All agreed upon points are implemented / addressed. Describe changes from pre-implementation design.
- Yes
Anything not implemented (discovered or discussed during work) has a follow-up story.
- No
Code is tested and passing, both automated and manual, what manual testing was done is described.
- Manual test performed by deploying a multi node cluster and confirming that everything got installed properly
New or changed UX/UX and has gotten feedback from stakeholders.
- N/A
Documentation has been updated.
- Yes
Behaves appropriately at the intended scale (describe intended scale).
- No change in scale
Considered backwards and forwards compatibility issues between client and server.
- N/A
Follows our coding standards and GUI style guidelines.
- N/A

Terraform changes include:
- Adding a new DNS name for the loki URL
- Creating a new S3 bucket to hold Loki data and access resources for it.
Installer changes include:
- Adding new nginx configurations for the loki URL
- Installing and configuring the Loki service in the monitoring role
- Installing and configuring the Grafana Alloy service on all nodes
- Installing the Loki related dashboard on Grafana

Actions

Copy link

#19

Updated by Peter Amstutz about 1 month ago

Target version changed from Development 2024-10-09 sprint to Development 2024-10-23 sprint

Actions

Copy link

#20

Updated by Brett Smith about 1 month ago

Lucas Di Pentima wrote in #note-18:

16417-installer-loki @ 84ba7b0687

LGTM, thanks.

Actions

Copy link

#21

Updated by Lucas Di Pentima about 1 month ago

Status changed from In Progress to Resolved

Applied in changeset arvados|f830a66beea08bb4edf35a34e9388b44d95f5545.

Actions

Copy link

#22

Updated by Lucas Di Pentima about 1 month ago

Status changed from Resolved to In Progress

Re-opening this as I've discovered a bug in the installer code that makes grafana installed on all nodes.

Actions

Copy link

#23

Updated by Lucas Di Pentima 30 days ago

Updates at c3e3074 - 16417-installer-loki-fixes branch
Test run: test-provision: #1012

Instead of requiring grafana-formula's managed package repo state, alloy_install state now manages it directly. This avoids having grafana installed on all nodes.
While manually testing a multi-node cluster deployment, Loki was failing to start because the auto-generated S3 bucket password included 2 / chars that cannot parse correctly in its config file. Now we urlencode() the password to fix it.

Actions

Copy link

#24