Idea #21646
closedInvestigate capabilities of logging systems
Description
Things we'd like to know:
- Structured logging / ability to search specific fields
- Size limits
- Performance (how many log entries per second can it handle)
- Remote capabilities (want to be able to send logs to a central host)
- How quickly do logs appear on the remote host
- Ease of integration with standard logging libraries we use for Go/Python
- CLI and web UI for doing queries
- Archiving / log rotation
- Integration with cloud provider (AWS) log management
- Ease of deployment
The main options we know of are syslog, journald and AWS logwatch
Updated by Lucas Di Pentima 6 months ago
Just want to nominate a project worth looking into, I've used it a long long time ago and it was great: https://www.fluentd.org
Updated by Peter Amstutz 3 months ago
Grafana Alloy:
https://grafana.com/docs/alloy/latest/introduction/
is a distribution of OpenTelemetry (OTel) Collector:
https://github.com/open-telemetry/opentelemetry-collector
It seems that Loki can accept logs it its own format a well as OTel, so presumably that Alloy uses.
It's unclear if Logrus has a hook for OTel. It does have one for Loki.
Updated by Peter Amstutz 3 months ago
Answering the question in the description for Loki:
- Structured logging / ability to search specific fields
yes
- Size limits
don't know
- Performance (how many log entries per second can it handle)
don't know
- Remote capabilities (want to be able to send logs to a central host)
Yes
- How quickly do logs appear on the remote host
don't know
- Ease of integration with standard logging libraries we use for Go/Python
Available for Go, haven't looked at Python yet.
- CLI and web UI for doing queries
yes. CLI client exist, WebUI is integrated with Grafana.
- Archiving / log rotation
yes, has retention rules https://grafana.com/docs/loki/latest/operations/storage/retention/
- Integration with cloud provider (AWS) log management
Seems like there's work on integration with OTel.
- Ease of deployment
Has published deb packages.