Feature #19982: Ability to know when a container died because of spot instance reclamation and option to resubmit - Arvados

Actions

Copy link

Feature #19982

open

Ability to know when a container died because of spot instance reclamation and option to resubmit

Added by Peter Amstutz 11 months ago. Updated 11 days ago.

Status:

In Progress

Priority:

Normal

Assigned To:

Alex Coleman

Category:

CWL

Target version:

Development 2024-01-03 sprint

Start date:

Due date:

% Done:

Estimated time:

(Total: 0.00 h)

Story points:

3.0

Description

New arvados-cwl-runner behavior when spot instances are enabled

When submitting spot instance, don't retry
Ability to detect when a container failed due to reclaimed spot instance (#19961)
Exit code to indicate workflow failed due to spot instance
Option to automatically re-submit as reserved instance

Subtasks 1 (1 open — 0 closed)

Related issues

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Blocked by Feature #19961: Detect and log spot instance interruption notices added

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Category changed from CWL to Crunch
Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Category changed from Crunch to CWL

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Story points set to 2.0

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Story points changed from 2.0 to 3.0

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Target version changed from To be groomed to To be scheduled

Actions

Copy link

Updated by Peter Amstutz 10 months ago

Related to Feature #19975: Option to re-submit container with higher memory request if previous job was killed and crunchstat shows >90% memory usage added

Actions

Copy link

#10

Updated by Peter Amstutz 10 months ago

Related to Feature #19974: Option to re-submit preemptible jobs to reserved nodes when previous attempt was interrupted added

Actions

Copy link

#11

Updated by Peter Amstutz 8 months ago

Related to Story #18179: Better spot instance support added

Actions

Copy link

#12

Updated by Peter Amstutz 5 months ago

Target version changed from To be scheduled to Development 2023-08-02 sprint

Actions

Copy link

#13

Updated by Peter Amstutz 5 months ago

Assigned To set to Alex Coleman

Actions

Copy link

#14

Updated by Peter Amstutz 4 months ago

Target version changed from Development 2023-08-02 sprint to Development 2023-08-16

Actions

Copy link

#15

Updated by Peter Amstutz 4 months ago

Target version changed from Development 2023-08-16 to Development 2023-08-30

Actions

Copy link

#16

Updated by Peter Amstutz 3 months ago

Target version changed from Development 2023-08-30 to Development 2023-09-13 sprint

Actions

Copy link

#17

Updated by Brett Smith 3 months ago

Related to Bug #20606: Unstartable preemptible:true containers should not be reused by non-retryable preemptible:false requests added

Actions

Copy link

#18

Updated by Brett Smith 3 months ago

We should consider undoing or narrowing the reuse changes we made in #20606 after we implement this. If Arvados gets better about retrying, then odds go up that the reuse narrowing is more likely to be wasteful than helpful.

Actions

Copy link

#19