Project

General

Profile

Actions

Feature #19320

closed

Get actual instance price information by calling AWS APIs

Added by Peter Amstutz over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
Story points:
2.0
Release relationship:
Auto

Description

This data structure would be populated at a configurable `ContainerCostUpdateInterval` (default every 2 minutes?) for running containers, as well as immediately at the start and the end of a container.

For AWS spot instances, we would have a goroutine that stores the relevant spot price history (DescribeSpotPriceHistory API call) for the configured availability zone in the database. Awkwardly we currently derive the availability zone from `Containers/CloudVMs/DriverParameters/SubnetID`, perhaps we should add an explicit value in the config file). The history would only be stored up to the oldest running job (and maybe while nothing is running, up to the current point in time?). It could be refreshed on the same frequency as `ContainerCostUpdateInterval`, but ideally we would stop updating when nothing is running.

For regular AWS instances, we can get the hourly price for all node types in our AZ, API call TBD. This could be updated once a day (configurable?).

For both spot and regular AWS instances, we also need to account for the cost of any extra attached EBS storage. API call TBD.

As a bonus, or a follow-on, it would also be useful for the container request record to contain:

  • the total cost to run the process tree (sum of the container and all child containers)
  • the incremental cost to run the process tree (this is the previous sum excluding container requests that reused existing containers)

This could be calculated by the API server or controller and added to the container request (maybe as a property) so that it is easily available for display in Workbench.

Follow on work, outside the scope of this ticket:
  • Azure support for regular and spot instances
  • Costanalyzer support to use this cost data
  • Workbench2 support to show this cost data live
  • Gather interesting statistics for spot instances that could inform scheduling decisions in the future (e.g. how frequently certain node types get evicted, or the average spot price by instance type over time)

Subtasks 1 (0 open1 closed)

Task #19905: Review 19320-spot-pricingResolvedTom Clegg01/23/2023Actions

Related issues

Related to Arvados Epics - Idea #18179: Better spot instance supportIn Progress03/01/202203/31/2024Actions
Related to Arvados - Feature #20040: Getting additional cost info from AWSNewTom CleggActions
Blocked by Arvados - Feature #18205: [api] [cloud] add compute instance price to container recordResolvedTom Clegg08/08/2022Actions
Actions

Also available in: Atom PDF