Actions
Feature #18180
closed
Ability to control use of spot instances on a per-workflow and step level
Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
Story points:
-
Release:
Release relationship:
Auto
Description
- container_request scheduling_parameter['preemptible'] doesn't have to match the setting of the cluster
- ability to choose spot instances when submitting workflow with arvados-cwl-runner
- --enable-preemptible --disable-preemptible
- gets passed through to the arvados-cwl-runner leader instance
- --disable-preemptible opts out for whole workflow (overrides)
- --enable-preemptible opts in for the whole workflow
- Individual steps can opt out if preemptible is enabled
Related issues
Updated by Peter Amstutz over 2 years ago
- Category set to Crunch
- Description updated (diff)
Updated by Peter Amstutz over 2 years ago
- Related to Idea #18179: Better spot instance support added
Updated by Peter Amstutz about 2 years ago
- Target version set to 2022-03-16 sprint
- Description updated (diff)
Updated by Peter Amstutz about 2 years ago
- Subject changed from Ability to control use of spot instances on a per-workflow basis to Ability to control use of spot instances on a per-workflow and step level
Updated by Peter Amstutz about 2 years ago
- Target version changed from 2022-03-16 sprint to 2022-03-30 Sprint
Updated by Peter Amstutz about 2 years ago
18180-cwl-spot @ 0ffd9efd1c9e088ec713e2303012463465008d40
- Adds --enable/disable-preemptible command line options
- Adds arv:UsePreemptible hint
- Adds tests
Updated by Lucas Di Pentima about 2 years ago
Re-running failed job: developer-run-tests-sdk-python-ruby: #2286
Updated by Lucas Di Pentima about 2 years ago
Code looks good to me. Here're my doc-related comments:
- Files
arv-cwl-schema*.yml: Thefields.class.dockeys seems to need fixing. (Copy&pasted content from other extension?) - Documentation at
/admin/spot-instances.htmlmight need some update: "When UsePreemptibleInstances is enabled, child containers (workflow steps) will automatically be made preemptible..." - I think the doc pages
/user/cwl/cwl-runner.html&/user/cwl/cwl-run-options.htmlcould also need some updates regarding this new feature.
Updated by Peter Amstutz about 2 years ago
Lucas Di Pentima wrote:
Code looks good to me. Here're my doc-related comments:
- Files
arv-cwl-schema*.yml: Thefields.class.dockeys seems to need fixing. (Copy&pasted content from other extension?)
Fixed.
- Documentation at
/admin/spot-instances.htmlmight need some update: "When UsePreemptibleInstances is enabled, child containers (workflow steps) will automatically be made preemptible..."
Added more text about it.
- I think the doc pages
/user/cwl/cwl-runner.html&/user/cwl/cwl-run-options.htmlcould also need some updates regarding this new feature.
Updated cwl-run-options and cwl-style.
While I was at it, I also improved the formatting of cwl-style a bit.
18180-cwl-spot @ f21f62d680e200d113d6fd700db743e702e96602
Updated by Ward Vandewege about 2 years ago
I noticed that the "Scheduling parameters" section at doc/api/methods/containers.html says
| key | type | description | notes |
| preemptible | boolean | If true, the dispatcher will ask for a preemptible cloud node instance (eg: AWS Spot Instance) to run this container. | Optional. Default is false. |
Isn't that default more subtle now, depending on the cluster configuration?
Actions