[Crunch] [UX] Improve log throttling
With the addition of log throttling (Exceeded rate 65536 bytes per 60 seconds (crunch_log_throttle_bytes). Logging will be silenced for the next 16 seconds.)
I can lose a lot of useful logs in that time window. My theory is to reduce the logging of crunchstat in order to see more what is going on in my computation without that clutter. If logging crunchstat does not add much to the rate, it might be useful to have a way to increase the crunch_log_throttle_bytes from an SDK or something.
Basically, I want a feature to be able to see all the log output without throttling (but I know that was impossible before crunch_log_throttle)
Implementation¶Log throttling only applies to live logs, not the logs stored in Keep. Improve the message logs when jobs get throttled to help clarify this.
- Before "line.replace message" in source:services/api/lib/crunch_dispatch.rb#L478:
message += " A complete log is still being written to Keep, and will be available when the job finishes.\n"
- Make sure
messagealways ends up with exactly one "\n" (currently there's a bit of inconsistency here).
- If no other line matching this description has been passed through for 5 seconds, update the "last non-terminal line segment propagated" timestamp for the job, and handle this line normally using the existing log throttle code.
- If a non-terminal segment has been passed through in the last 5 seconds:
- suppress this line and don't count it toward the throttle counters.
- if this is the first time skipping a non-terminal line segment, replace the log line with an appropriate message and append the "complete log" reassurance above, as with the other messages.
Rate-limiting partial segments of long lines to one every 5 seconds.
"5" should be in a new config var