Project

General

Profile

Feature #8019

Updated by Peter Amstutz over 7 years ago

Implement log throttling in crunch-run.    Behavior should be equivalent to services/api/lib/crunch_dispatch.rb#rate_limit 

 Use same configuration parameters as currently in API server config: 

 <pre> 
   # These two settings control how frequently log events are flushed to crunchrun support for the 
   following parameters:  
 - limit on # database.    Log of lines are buffered until either crunch_log_bytes_per_event 
   # has been reached or crunch_log_seconds_between_events has elapsed since 
   # the last flush. 
   crunch_log_bytes_per_event: 4096 
   crunch_log_seconds_between_events: 1 

   # The sample per period for throttling logs, in seconds. 
   crunch_log_throttle_period: 60 

   # Maximum number 
 - length of bytes that job can log over crunch_log_throttle_period 
   period 
 - total # before being silenced until the end of the period. 
   crunch_log_throttle_bytes: 65536 

   # Maximum number of lines that per job can log over crunch_log_throttle_period 
   # before being silenced until the end 

 Use same of parameters as used by the period. 
   crunch_log_throttle_lines: 1024 

   # Maximum bytes that may be logged by a single job.    Log bytes that are 
   # silenced by throttling are not counted against this total. 
   crunch_limit_log_bytes_per_job: 67108864 
 </pre> 

 Above parameters should be API server today, but published in the discovery discover document for use by crunch-run. 
 so that crunchrun can query can

Back