Bug #17775

[a-d-c] the user should be able to see when preemptible nodes get shut down and the running container requeued

Added by Ward Vandewege 3 months ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

When a preemptible node is shut down by the cloud provider while running a workflow, the container is automatically requeued. From the user's perspective, it looks like a container is runnning and then suddenly disappears and gets replaced with a new one that in the 'queued' state.

The user should be able to see what happened ("container was stopped and requeued because the cloud node failed/preemptible instance was stopped"). More generally, it would be nice if for a given CR, the user could see all the containers that were created/started/failed (and the relevant timestamps) for that CR in wb/wb2. Right now, when a container fails and automatically requeued, that is quite hard to see.

The cloud providers send a signal when a preemptible node is going to be shut down; ideally we'd catch that and log it, and bubble that up to the user.

History

#1 Updated by Ward Vandewege 3 months ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 2 months ago

  • Target version deleted (To Be Groomed)

Also available in: Atom PDF