Feature #7399
Updated by Tom Clegg over 8 years ago
h2. Bug report With the addition of log throttling (Exceeded rate 65536 bytes per 60 seconds (crunch_log_throttle_bytes). Logging will be silenced for the next 16 seconds.) I can lose a lot of useful logs in that time window. My theory is to reduce the logging of crunchstat in order to see more what is going on in my computation without that clutter. If logging crunchstat does not add much to the rate, it might be useful to have a way to increase the crunch_log_throttle_bytes from an SDK or something. Basically, I want a feature to be able to see all the log output without throttling (but I know that was impossible before crunch_log_throttle) h2. Implementation Log throttling only applies to live logs, not the logs stored in Keep. Improve the message logs when jobs get throttled to help clarify this. * Before "line.replace message" in crunch-dispatch.rb line 518: * <pre><code class="ruby"> message += " A "A complete log is still being written to Keep, and will be available when the job finishes.\n" </code></pre> * Make sure @message@ always ends up with exactly one "\n" (currently there's a bit of inconsistency here). When the log throttle is open and crunch-dispatch reads a log line that has a crunch-job "stderr" tag and content that *begins and ends* with "[...]" -- i.e., is the middle of a long line that has been split up by crunchstat -- handle it specially: * If no other line matching this description has been passed through for 5 seconds, update the "last non-terminal line segment propagated" timestamp for the job, and handle this line normally using the existing log throttle code. * If a non-terminal segment _has_ been passed through in the last 5 seconds: ** suppress this line and don't count it toward the throttle counters. ** if this is the first time skipping a non-terminal line segment, replace the log line with an appropriate message and append the "complete log" reassurance above, as with the other messages. ** <pre>Rate-limiting partial segments of long lines to one every 5 seconds.</pre> "5" should be in a new config var @Rails.configuration.crunch_log_partial_line_throttle_period@.