[Crunch2] Logs should be saved to disk when container is cancelled
crunch-run tries to save a log file after the container ends, regardless of final state, but (sometimes?) this doesn't work. Example: su92l-xvhdp-4j98m0zgu9xst51Some possible explanations:
- crunch-dispatch-slurm cancels the slurm job as soon as it notices the container is cancelled. crunch-run catches SIGTERM and tries to write the buffered output and logs, but (according to sample logs) seems to give up 30-40 seconds later without actually writing them.
- even if crunch-run gets that far, it seems apiserver would refuse to update the output or log field of a container whose state is Cancelled.