Feature #18205

[api] [cloud] add live compute instance price to container record

Added by Ward Vandewege 2 months ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

This applies to cloud installations only.

Add a data structure to the container record that contains:

  • the total cost of a container (if the container is running, the cost up to this point in time)
  • the current hourly cost of that container, if it is not complete yet (after it is complete, it could be updated to the average hourly cost over the duration of the container)

This data structure would be populated at a configurable `ContainerCostUpdateInterval` (default every 2 minutes?) for running containers, as well as immediately at the start and the end of a container.

For AWS spot instances, we would have a goroutine that stores the relevant spot price history (DescribeSpotPriceHistory API call) for the configured availability zone in the database. Awkwardly we currently derive the availability zone from `Containers/CloudVMs/DriverParameters/SubnetID`, perhaps we should add an explicit value in the config file). The history would only be stored up to the oldest running job (and maybe while nothing is running, up to the current point in time?). It could be refreshed on the same frequency as `ContainerCostUpdateInterval`, but ideally we would stop updating when nothing is running.

For regular AWS instances, we can get the hourly price for all node types in our AZ, API call TBD. This could be updated once a day (configurable?).

For both spot and regular AWS instances, we also need to account for the cost of any extra attached EBS storage. API call TBD.

Follow on work, outside the scope of this ticket:
  • Azure support for regular and spot instances
  • Costanalyzer support to use this cost data
  • Workbench2 support to show this cost data live
  • Gather interesting statistics for spot instances that could inform scheduling decisions in the future (e.g. how frequently certain node types get evicted, or the average spot price by instance type over time)

Related issues

Blocks Arvados Epics - Story #18179: Better spot instance supportNew05/01/202207/31/2022

Blocks Arvados - Feature #17695: [costanalyzer] make an accurate report for spot instances on AWSIn Progress

History

#1 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#2 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#3 Updated by Ward Vandewege 2 months ago

  • Description updated (diff)

#4 Updated by Ward Vandewege 2 months ago

#5 Updated by Ward Vandewege 2 months ago

  • Blocks Feature #17695: [costanalyzer] make an accurate report for spot instances on AWS added

Also available in: Atom PDF