Bug #3769
closed[API] In crunch-dispatch, throttle by bytes_per_minute or _node_minute
Description
crunch_limit_log_event_bytes_per_job is currently a hard-coded number in the server config. It would be much more useful to have a number that is proportional to the number of task hours in the job. That way, long-running or big jobs are allowed more log output.
If that's easier, adding a crunch_limit_log_event_bytes_per_task field could be good enough, then I could leave crunch_limit_log_event_bytes_per_job unlimited (or very high).
Implementation:- Add a crunch_limit_log_event_bytes_per_minute config to
services/api/config/application.default.yml
- In
services/api/script/crunch-dispatch.rb
, temporarily stop propagating log messages to the logs table when the per-minute threshold is exceeded.- This doesn't have to be perfect: dividing the job time into 60-second chunks and throttling each chunk independently should be good enough, even though it means a job can exceed the threshold between t=30s and t=90s. (At worst, a job can exceed the threshold by a factor of 2 this way, and can't exceed it continuously.)
- When the limit is hit, emit "[log messages suppressed for N seconds]" to the logs table (N = seconds until next 60-second interval starts).
- When resuming logging, emit "[N bytes of log messages skipped]" to the logs table.
Related issues
Updated by Peter Amstutz over 9 years ago
- Target version set to Arvados Future Sprints
Updated by Tom Clegg over 9 years ago
- Subject changed from [API] crunch_limit_log_event_bytes_per_job is not smart enough to [API] In crunch-dispatch, throttle by bytes_per_minute or _node_minute
- Description updated (diff)
- Category set to Crunch
Updated by Tom Clegg over 9 years ago
- Target version changed from Arvados Future Sprints to 2014-10-08 sprint
Updated by Peter Amstutz over 9 years ago
Should it print "log messages silenced" every 60 seconds as long is the rate is being exceeded, or only when at the start?
Updated by Peter Amstutz over 9 years ago
- Status changed from New to In Progress
Updated by Ward Vandewege over 9 years ago
services/api/config/application.default.yml
+ # database. Logs lines are buffered until either crunch_log_bytes_per_event
Make that "Log lines".
+ # has been reached or crunch_log_seconds_between_events has ellapsed since
One l in elapsed.
in services/api/script/crunch-dispatch.rb
+ #puts "Handle line at #{now - running_job[:log_throttle_timestamp]}, buf bytes #{line.size}, so far #{running_job[:log_throttle_bytes_so_far]}, throttled #{running_job[:log_throttl
+
That line can go I think?
Other than that, the code looks like it should do what the description says. I assume you've tested this locally?
Updated by Anonymous over 9 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|commit:60998a3875f79482533976e6e0ee0f99a9589c46.