Arvados

Just want to nominate a project worth looking into, I've used it a long long time ago and it was great: https://www.fluentd.org

Grafana Alloy:

https://grafana.com/docs/alloy/latest/introduction/

is a distribution of OpenTelemetry (OTel) Collector:

https://github.com/open-telemetry/opentelemetry-collector

It seems that Loki can accept logs it its own format a well as OTel, so presumably that Alloy uses.

It's unclear if Logrus has a hook for OTel. It does have one for Loki.

https://github.com/sirupsen/logrus/wiki/Hooks

Answering the question in the description for Loki:

• Structured logging / ability to search specific fields

yes

• Size limits

don't know

• Performance (how many log entries per second can it handle)

don't know

• Remote capabilities (want to be able to send logs to a central host)

Yes

• How quickly do logs appear on the remote host

don't know

• Ease of integration with standard logging libraries we use for Go/Python

Available for Go, haven't looked at Python yet.

• CLI and web UI for doing queries

yes. CLI client exist, WebUI is integrated with Grafana.

• Archiving / log rotation

yes, has retention rules https://grafana.com/docs/loki/latest/operations/storage/retention/

• Integration with cloud provider (AWS) log management

Seems like there's work on integration with OTel.

• Ease of deployment

Has published deb packages.

Need to address access control.