Bug #3769
Updated by Tom Clegg over 9 years ago
crunch_limit_log_event_bytes_per_job is currently a hard-coded number in the server config. It would be much more useful to have a number that is proportional to the number of task hours in the job. That way, long-running or big jobs are allowed more log output.
If that's easier, adding a crunch_limit_log_event_bytes_per_task field could be good enough, then I could leave crunch_limit_log_event_bytes_per_job unlimited (or very high).
Implementation:
* Add a crunch_limit_log_event_bytes_per_minute config to @services/api/config/application.default.yml@
* In @services/api/script/crunch-dispatch.rb@, _temporarily_ stop propagating log messages to the logs table when the per-minute threshold is exceeded.
** This doesn't have to be perfect: dividing the job time into 60-second chunks and throttling each chunk independently should be good enough, even though it means a job can exceed the threshold between t=30s and t=90s. (At worst, a job can exceed the threshold by a factor of 2 this way, and can't exceed it continuously.)
** When the limit is hit, emit "[log messages suppressed for N seconds]" to the logs table (N = seconds until next 60-second interval starts).
** When resuming logging, emit "[N bytes of log messages skipped]" to the logs table.