Project

General

Profile

Actions

Feature #17695

open

[costanalyzer] make an accurate report for spot instances on AWS

Added by Ward Vandewege about 1 year ago. Updated 10 months ago.

Status:
In Progress
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

It looks like AWS has a "DescribeSpotPriceHistory" API:

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSpotPriceHistory.html
https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-spot-price-history.html

That could be used to calculate the actual spot price for a container marked as run on an "Preemptible" node. We'd have to make sure to only use this code if we're on AWS.


Related issues

Related to Arvados Epics - Story #18179: Better spot instance supportNew03/01/202207/31/2022

Actions
Blocked by Arvados - Feature #18205: [api] [cloud] add live compute instance price to container recordNew

Actions
Actions #1

Updated by Ward Vandewege about 1 year ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege 10 months ago

  • Description updated (diff)
Actions #3

Updated by Ward Vandewege 10 months ago

  • Related to Story #18179: Better spot instance support added
Actions #4

Updated by Ward Vandewege 10 months ago

  • Status changed from New to In Progress

I have a proof of concept that works, however:

  • AWS credentials are required to get results
  • the region/az needs to be specified, and we don't keep that information (it's not in node.json, the az is not even in our config, it's derived from the SubnetID)
  • the spot price history only goes back 2 months (established empirically, I have not found this in any docs yet)

What this means is that we should add this functionality (calculating the cost of the job) to a-d-c or crunch-run, and that it should be added to the node.json file in the log collection. Costanalyzer can then just report that cost number directly, without having to do AWS api calls. And perhaps more importantly, we could also display it in Workbench2. Should we record it in the container record?

Update: we wrote this up in #18205

Actions #5

Updated by Ward Vandewege 9 months ago

  • Blocked by Feature #18205: [api] [cloud] add live compute instance price to container record added
Actions

Also available in: Atom PDF