Project

General

Profile

Actions

Bug #8811

closed

[Crunch] `srun --nodes=1` reports "Unable to create job step: Required node not available (down or drained)" and exits 1

Added by Brett Smith over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
0.5

Description

Bug report

In a job that only had one node allocated to it:

2016-03-26_03:24:57 wx7k5-8i9sb-1wvsbbyc8vsteeb 24686  load docker image: start
2016-03-26_03:24:57 wx7k5-8i9sb-1wvsbbyc8vsteeb 24686  stderr starting: ['srun','--nodelist=compute3','/bin/bash','-o','pipefail','-ec',' if ! docker.io images -q --no-trunc --all | grep -qxF b85dffb1be2ca7bc757be6ff8ae4873a45214918282ef42cc2cbc2cead63356b; then     arv-get 877b0bb063029e309ec3d0624e75eeda\\+503\\/b85dffb1be2ca7bc757be6ff8ae4873a45214918282ef42cc2cbc2cead63356b\\.tar | docker.io load fi ']
2016-03-26_03:29:56 wx7k5-8i9sb-1wvsbbyc8vsteeb 24686  load docker image: exit 0
2016-03-26_03:29:56 wx7k5-8i9sb-1wvsbbyc8vsteeb 24686  check --memory-swap feature: start
2016-03-26_03:29:56 wx7k5-8i9sb-1wvsbbyc8vsteeb 24686  stderr starting: ['srun','--nodes=1','docker.io','run','--help']
2016-03-26_03:29:56 wx7k5-8i9sb-1wvsbbyc8vsteeb 24686  stderr srun: error: Unable to create job step: Required node not available (down or drained)
2016-03-26_03:29:56 wx7k5-8i9sb-1wvsbbyc8vsteeb 24686  check --memory-swap feature: exit 1

This should be detected and treated the same way as any other node failure, by exiting so that crunch-dispatch retries the job.

Fix

Near the end of srun_sync, check if $main::please_freeze or $Jobstep->{tempfail} (you might need to refer to it by a different name) are true. If either of them are true, and $exited is 0, coerce $exited to be 255, so that the caller uses its normal error handling code to deal with the problem.


Subtasks 1 (0 open1 closed)

Task #8862: Review 8811-srun-sync-tempfail-wipResolvedPeter Amstutz03/31/2016Actions

Related issues

Related to Arvados - Bug #8810: [Crunch] `docker load` fails to connect to endpoint; srun exits 0ResolvedBrett Smith04/05/2016Actions
Actions

Also available in: Atom PDF