Arvados

crunch_limit_log_event_bytes_per_job is currently a hard-coded number in the server config. It would be much more useful to have a number that is proportional to the number of task hours in the job. That way, long-running or big jobs are allowed more log output.

If that's easier, adding a crunch_limit_log_event_bytes_per_task field could be good enough, then I could leave crunch_limit_log_event_bytes_per_job unlimited (or very high).

• Add a crunch_limit_log_event_bytes_per_minute config to services/api/config/application.default.yml
• In services/api/script/crunch-dispatch.rb, temporarily stop propagating log messages to the logs table when the per-minute threshold is exceeded.
  • This doesn't have to be perfect: dividing the job time into 60-second chunks and throttling each chunk independently should be good enough, even though it means a job can exceed the threshold between t=30s and t=90s. (At worst, a job can exceed the threshold by a factor of 2 this way, and can't exceed it continuously.)
  • When the limit is hit, emit "[log messages suppressed for N seconds]" to the logs table (N = seconds until next 60-second interval starts).
  • When resuming logging, emit "[N bytes of log messages skipped]" to the logs table.