Bug #9688

[Crunch2] Limit number of dispatch attempts per container

Added by Tom Clegg over 2 years ago. Updated 12 days ago.

Status:
Duplicate
Priority:
High
Assigned To:
-
Category:
-
Target version:
Start date:
08/02/2016
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Problem

There are circumstances where crunch-dispatch-* tries to run a container, but something fails before the container gets to Running state, so the container goes back to Queued state. See #9679.

If the same problem keeps happening, the container just flaps between Queued and Locked.

After a certain amount of time, or number of retries, we should really just give up and cancel the container.

Proposed solutions

Pick one of:
  • (crunch-dispatch-*) If a single container gets dispatched more than N times (over a period of at least M seconds) by a single crunch-dispatch-* process, but still won't run, give up and change state to Cancelled.
  • (API server) If a container has been Locked and returned to Queued state, and is more than M seconds old, cancel it.
Also (optional), to mitigate starvation risk in a multiple-dispatch setup:
  • Introduce a delay between "return container X to queue" and "re-attempt X".

Related issues

Related to Arvados - Bug #9679: [Crunch2] Provide feedback when a container is submitted to slurm but does not runResolved2016-07-29

Has duplicate Arvados - Bug #14540: [API] Limit number of container lock/unlock cyclesDuplicate

Has duplicate Arvados - Bug #11561: [API] Limit number of lock/unlock cycles for a given containerNew2017-04-26

History

#1 Updated by Peter Amstutz 19 days ago

  • Related to Bug #11561: [API] Limit number of lock/unlock cycles for a given container added

#2 Updated by Tom Morris 19 days ago

  • Priority changed from Normal to High
  • Target version set to To Be Groomed

#3 Updated by Tom Morris 19 days ago

  • Related to Bug #14540: [API] Limit number of container lock/unlock cycles added

#4 Updated by Peter Amstutz 12 days ago

  • Related to deleted (Bug #14540: [API] Limit number of container lock/unlock cycles)

#5 Updated by Peter Amstutz 12 days ago

  • Has duplicate Bug #14540: [API] Limit number of container lock/unlock cycles added

#6 Updated by Peter Amstutz 12 days ago

  • Related to deleted (Bug #11561: [API] Limit number of lock/unlock cycles for a given container)

#7 Updated by Peter Amstutz 12 days ago

  • Has duplicate Bug #11561: [API] Limit number of lock/unlock cycles for a given container added

#8 Updated by Peter Amstutz 12 days ago

  • Status changed from New to Duplicate

Also available in: Atom PDF