Bug #7263
Updated by Brett Smith over 8 years ago
h2. Original bug report
c97qk-8i9sb-rjmjzrz54gf61os on c97qk was cancelled, but crunch-job doesn't notice and lets the job continue running.
We also saw this recently with a job on su92l.
h2. Diagnosis
The main loop of crunch-job has this code:
<pre> my $gotsome
= readfrompipes ()
+ reapchildren ();
if (!$gotsome)
{
check_refresh_wanted();
check_squeue();
update_progress_stats();
select (undef, undef, undef, 0.1);
}
</pre>
@check_refresh_wanted@ is the function that will notice if the job has been canceled and the child processes signaled accordingly. If the cancel comes in while we're in this loop, we're unlikely to notice it as long as we keep receiving data from the child processes—which seems pretty likely now that we have crunchstat in the mix.
This diagnosis is not a 100% sure thing, because the pipes we're reading from are opened in nonblocking mode. So they would have to be very busy to keep avoiding this branch. But it's at least still possible, particularly since this problem seems to strike jobs that have many parallel tasks.
h2. Fix
Suggest checking refresh_trigger (and the other things) every ~two seconds, even during times when @readfrompipes()@ and @reapchildren()@ are keeping us 100% busy.
<pre><code class="diff">
- if (!$gotsome) {
+ if (!$gotsome || $latest_refresh + 2 < scalar time) {
</code></pre>
After this is merged and deployed, test whether you can successfully cancel massively parallel jobs. If it works consistently, I think we can mark this ticket resolved. Otherwise, we'll need to do another pass on this. should call @check_refresh_wanted@ more regularly, regardless of what the child processes are doing.