Arvados

We've established that Journald simply can not handle the amount of logging output some of our services produce (e.g. arvados-controller).

There are of course ways around that, e.g. on our high volume sites we have a systemd override file like this:

[Service]
Type=simple
ExecStart=
ExecStart=/bin/bash -ce "exec /usr/bin/arvados-controller 2>&1 | /usr/bin/rotatelogs -f /var/log/arvados/%%Y%%m%%d-arvados-controller.log 86400" 

It would be nice to build this into arvados-server. It looks like this library might work with some tweaks: https://github.com/natefinch/lumberjack.

The idea is to default to logging as we do now, but by setting a logging directory value in the config file, we would start logging date-stamped files in that directory and auto-rotating them (configurable) after a default of 30 days.