[Crunch2] Limit number of dispatch attempts per container
There are circumstances where crunch-dispatch-* tries to run a container, but something fails before the container gets to Running state, so the container goes back to Queued state. See #9679.
If the same problem keeps happening, the container just flaps between Queued and Locked.
After a certain amount of time, or number of retries, we should really just give up and cancel the container.
Proposed solutions¶Pick one of:
- (crunch-dispatch-*) If a single container gets dispatched more than N times (over a period of at least M seconds) by a single crunch-dispatch-* process, but still won't run, give up and change state to Cancelled.
- (API server) If a container has been Locked and returned to Queued state, and is more than M seconds old, cancel it.
- Introduce a delay between "return container X to queue" and "re-attempt X".