Arvados

Note the large time gap between the Docker error and the node failure. That seems suspicious.

See 86f7740, which might fix this.

Need to check the Docker API docs: is the failing command the one in the if condition? If so, that would explain why it exits 0. We could add else; exit ${PIPESTATUS[0]} to return the exit code of docker images.

We have this

if ! docker images | grep
then
  arv-get | docker load
fi

If docker images fails, then we end up calling arv-get. I suspect this is incorrect. We should probably abort if docker images fails: we're probably just wasting our time running docker load.

We probably want something more like this:

if docker images | grep; then
  true
elif [[ ${PIPESTATUS[0]} == 0 ]]; then
  arv-get | docker load
else
  exit ${PIPESTATUS[0]}
fi

However, I don't think this is the bug we're seeing here.

The "node failure" and "job step aborted" messages suggest slurm (not our shell script) is behaving badly by exiting 0.

1. docker images failed, and exited non-zero ...so docker images | grep failed ...so we tried docker load
2. docker load succeeded, and exited zero
3. Node failed during srun teardown
4. Slurm printed error messages about the node failure
5. Slurm propagated the docker load exit status
6. crunch-job asked slurm to do other things with the node
7. of course the other things failed because the node had failed

In this case, the bug amounts to "if a node fails between times A and B, we get "node failure" error messages before the node failure impacts our ability to run programs on that node". In that case, it seems much less worrisome than "exited 0 on failure".

Tom Clegg wrote:

We probably want something more like this:

[snip]

The branch I have up for review implements that.

In this case, the bug amounts to "if a node fails between times A and B, we get "node failure" error messages before the node failure impacts our ability to run programs on that node". In that case, it seems much less worrisome than "exited 0 on failure".

I believe we changed our behavior here in 86f7740. Now crunch-job will "see" the node failure message, and treat the setup as failed as a result. This seems like an improvement to me. If a node failure happened during a setup task, things are unlikely to go well when we start dispatching actual tasks to it. Better to fail as soon as we see trouble, to save time and avoid making logs more confusing with additional error messages.

8810-crunch-improve-docker-loading-wip @ df1447d LGTM. Thanks.

86f7740

Indeed, that seems to have already fixed the "confusing sequence of error messages" aspect. If I'm following the timing properly, #8810 and #8811 are different problems arising from the same underlying bug, "failed to notice early signs that the node is broken, and failed the job as a result", and both are fixed by the 8811 branch. And 8810 happened to help us see & fix another bug.

Tom Clegg wrote:

Indeed, that seems to have already fixed the "confusing sequence of error messages" aspect. If I'm following the timing properly, #8810 and #8811 are different problems arising from the same underlying bug, "failed to notice early signs that the node is broken, and failed the job as a result", and both are fixed by the 8811 branch. And 8810 happened to help us see & fix another bug.

Yeah, Peter made a similar comment at last week's backlog grooming, so I think we're all agreed on that now. Thanks.