Project

General

Profile

Actions

Bug #12893

closed

[Crunch2] Logs should be saved to disk when container is cancelled

Added by Tom Clegg about 6 years ago. Updated about 6 years ago.

Status:
Duplicate
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Story points:
-

Description

crunch-run tries to save a log file after the container ends, regardless of final state, but (sometimes?) this doesn't work. Example: su92l-xvhdp-4j98m0zgu9xst51

Some possible explanations:
  • crunch-dispatch-slurm cancels the slurm job as soon as it notices the container is cancelled. crunch-run catches SIGTERM and tries to write the buffered output and logs, but (according to sample logs) seems to give up 30-40 seconds later without actually writing them.
  • even if crunch-run gets that far, it seems apiserver would refuse to update the output or log field of a container whose state is Cancelled.

Related issues

Is duplicate of Arvados - Bug #12891: [crunch2] log collection not saved for cancelled jobResolvedTom Clegg01/22/2018Actions
Actions #1

Updated by Tom Morris about 6 years ago

  • Is duplicate of Bug #12891: [crunch2] log collection not saved for cancelled job added
Actions #2

Updated by Tom Morris about 6 years ago

  • Status changed from New to Duplicate
Actions #3

Updated by Peter Amstutz about 6 years ago

Yes, it may be that crunch-run is doing the right thing but the API server is refusing to allow any updates once the container has gone into a final state. So perhaps that needs to be relaxed to "allow setting the log field if it is null".

Actions

Also available in: Atom PDF