Feature #7399


[Crunch] [UX] Improve log throttling

Added by Bryan Cosca over 7 years ago. Updated over 7 years ago.

Assigned To:
Radhika Chippada
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)
Story points:


Bug report

With the addition of log throttling (Exceeded rate 65536 bytes per 60 seconds (crunch_log_throttle_bytes). Logging will be silenced for the next 16 seconds.)

I can lose a lot of useful logs in that time window. My theory is to reduce the logging of crunchstat in order to see more what is going on in my computation without that clutter. If logging crunchstat does not add much to the rate, it might be useful to have a way to increase the crunch_log_throttle_bytes from an SDK or something.

Basically, I want a feature to be able to see all the log output without throttling (but I know that was impossible before crunch_log_throttle)


Log throttling only applies to live logs, not the logs stored in Keep. Improve the message logs when jobs get throttled to help clarify this.
  • Before "line.replace message" in source:services/api/lib/crunch_dispatch.rb#L478:
  • message += " A complete log is still being written to Keep, and will be available when the job finishes.\n" 
  • Make sure message always ends up with exactly one "\n" (currently there's a bit of inconsistency here).
When the log throttle is open and crunch-dispatch reads a log line that has a crunch-job "stderr" tag and content that begins and ends with "[...]" -- i.e., is the middle of a long line that has been split up by crunchstat -- handle it specially:
  • If no other line matching this description has been passed through for 5 seconds, update the "last non-terminal line segment propagated" timestamp for the job, and handle this line normally using the existing log throttle code.
  • If a non-terminal segment has been passed through in the last 5 seconds:
    • suppress this line and don't count it toward the throttle counters.
    • if this is the first time skipping a non-terminal line segment, replace the log line with an appropriate message and append the "complete log" reassurance above, as with the other messages.
    • Rate-limiting partial segments of long lines to one every 5 seconds.

"5" should be in a new config var Rails.configuration.crunch_log_partial_line_throttle_period.

Subtasks 1 (0 open1 closed)

Task #8472: Review branch: 7399-log-throttlingResolvedRadhika Chippada09/29/2015


Related issues

Related to Arvados - Story #8552: [Crunch] Log throttling shouldn't throttle logs from crunchstat, crunch-jobClosed


Also available in: Atom PDF