Bug #9688

[Crunch2] Limit number of dispatch attempts per container

Added by Tom Clegg almost 3 years ago. Updated 3 months ago.

Status:
Duplicate
Priority:
High
Assigned To:
-
Category:
-
Target version:
-
Start date:
08/02/2016
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Problem

There are circumstances where crunch-dispatch-* tries to run a container, but something fails before the container gets to Running state, so the container goes back to Queued state. See #9679.

If the same problem keeps happening, the container just flaps between Queued and Locked.

After a certain amount of time, or number of retries, we should really just give up and cancel the container.

Proposed solutions

Pick one of:
  • (crunch-dispatch-*) If a single container gets dispatched more than N times (over a period of at least M seconds) by a single crunch-dispatch-* process, but still won't run, give up and change state to Cancelled.
  • (API server) If a container has been Locked and returned to Queued state, and is more than M seconds old, cancel it.
Also (optional), to mitigate starvation risk in a multiple-dispatch setup:
  • Introduce a delay between "return container X to queue" and "re-attempt X".

Related issues

Related to Arvados - Bug #9679: [Crunch2] Provide feedback when a container is submitted to slurm but does not runResolved07/29/2016

Has duplicate Arvados - Bug #14540: [API] Limit number of container lock/unlock cyclesDuplicate

Has duplicate Arvados - Bug #11561: [API] Limit number of lock/unlock cycles for a given containerResolved04/26/2017

History

#1 Updated by Peter Amstutz 5 months ago

  • Related to Bug #11561: [API] Limit number of lock/unlock cycles for a given container added

#2 Updated by Tom Morris 5 months ago

  • Priority changed from Normal to High
  • Target version set to To Be Groomed

#3 Updated by Tom Morris 5 months ago

  • Related to Bug #14540: [API] Limit number of container lock/unlock cycles added

#4 Updated by Peter Amstutz 4 months ago

  • Related to deleted (Bug #14540: [API] Limit number of container lock/unlock cycles)

#5 Updated by Peter Amstutz 4 months ago

  • Has duplicate Bug #14540: [API] Limit number of container lock/unlock cycles added

#6 Updated by Peter Amstutz 4 months ago

  • Related to deleted (Bug #11561: [API] Limit number of lock/unlock cycles for a given container)

#7 Updated by Peter Amstutz 4 months ago

  • Has duplicate Bug #11561: [API] Limit number of lock/unlock cycles for a given container added

#8 Updated by Peter Amstutz 4 months ago

  • Status changed from New to Duplicate

#9 Updated by Tom Morris 3 months ago

  • Target version deleted (To Be Groomed)

Also available in: Atom PDF