Project

General

Profile

Bug #11561

Updated by Peter Amstutz about 5 years ago

Currently, if a container cannot be started due to some infrastructure problem (whether or not it's related to the specific container) it will be retried repeatedly forever. 

 Proposed solution: 

 Add a site config knob (analogous to num_retries) that limits the number of times a container can be unlocked (moved from Locked to Queued state) before being automatically cancelled. 

 Add: 
 * Config key max_container_dispatch_attempts (default 5) 
 * DB column "lock_count" (do not include in API response) 
 * Increment lock_count during lock() 
 * When unlocking a container, if lock_count >= Rails.configuration.max_container_dispatch_attempts, change state to Cancelled instead of Queued (the unlock API should still respond 200 in this case) and update runtime_status[error] with an error message. 

 Write tests and update documentation.

Back