Arvados

Bug #7263

Updated by Brett Smith about 9 years ago

h2. Original bug report 

 c97qk-8i9sb-rjmjzrz54gf61os on c97qk was cancelled, but crunch-job doesn't notice and lets the job continue running. 

 We also saw this recently with a job on su92l. 

 h2. Diagnosis 

 The main loop of crunch-job has this code: 

 <pre>      my $gotsome 
	 = readfrompipes () 
	 + reapchildren (); 
     if (!$gotsome) 
     { 
       check_refresh_wanted(); 
       check_squeue(); 
       update_progress_stats(); 
       select (undef, undef, undef, 0.1); 
     } 
 </pre> 

 @check_refresh_wanted@ is the function that will notice if the job has been canceled and the child processes signaled accordingly.    If the cancel comes in while we're in this loop, we're unlikely to notice it as long as we keep receiving data from the child processes—which seems pretty likely now that we have crunchstat in the mix. 

 h2. Fix 

 I think we should call @check_refresh_wanted@ more regularly, regardless of what the child processes are doing.

Back