[servers] add logging with automatic rotation to user-specified directory
We've established that Journald simply can not handle the amount of logging output some of our services produce (e.g. arvados-controller).
There are of course ways around that, e.g. on our high volume sites we have a systemd override file like this:
[Service] Type=simple ExecStart= ExecStart=/bin/bash -ce "exec /usr/bin/arvados-controller 2>&1 | /usr/bin/rotatelogs -f /var/log/arvados/%%Y%%m%%d-arvados-controller.log 86400"
It would be nice to build this into arvados-server. It looks like this library might work with some tweaks: https://github.com/natefinch/lumberjack.
The idea is to default to logging as we do now, but by setting a logging directory value in the config file, we would start logging date-stamped files in that directory and auto-rotating them (configurable) after a default of 30 days.