Project

General

Profile

Actions

Feature #18205

closed

[api] [cloud] add compute instance price to container record

Added by Ward Vandewege over 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Description

Add new fields to the container record that contain:

  • the total cost of a container (if the container is running, the cost up to this point in time)
  • the total cost of the container and all of its child containers (for workflow runners) (if the container is running, the cost up to this point)
  • the incremental cost of the process tree, where reused containers are excluded from the calculation

When the container is completed, either the dispatcher or API server should compute the cost using InstanceType.Price. Getting live cloud prices will done in #19320


Subtasks 1 (0 open1 closed)

Task #19289: Review 18205-container-costResolvedTom Clegg08/08/2022Actions

Related issues

Blocks Arvados Epics - Idea #18179: Better spot instance supportIn Progress03/01/202206/30/2024Actions
Blocks Arvados - Feature #17695: [costanalyzer] make an accurate report for spot instances on AWSNewActions
Blocks Arvados - Feature #19319: Display container price informationResolvedStephen Smith10/31/2022Actions
Blocks Arvados - Feature #19320: Get actual instance price information by calling AWS APIsResolvedTom Clegg01/23/2023Actions
Actions #1

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #3

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #4

Updated by Ward Vandewege over 2 years ago

  • Blocks Idea #18179: Better spot instance support added
Actions #5

Updated by Ward Vandewege over 2 years ago

  • Blocks Feature #17695: [costanalyzer] make an accurate report for spot instances on AWS added
Actions #6

Updated by Peter Amstutz almost 2 years ago

  • Target version set to 2022-07-20
Actions #7

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-07-20 to 2022-08-17 sprint
Actions #8

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-08-17 sprint to 2022-08-03 Sprint
Actions #9

Updated by Peter Amstutz almost 2 years ago

  • Assigned To set to Tom Clegg
Actions #10

Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)
Actions #11

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-08-03 Sprint to 2022-08-17 sprint
Actions #12

Updated by Peter Amstutz almost 2 years ago

Actions #13

Updated by Peter Amstutz almost 2 years ago

  • Blocks Feature #19320: Get actual instance price information by calling AWS APIs added
Actions #14

Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)
  • Subject changed from [api] [cloud] add live compute instance price to container record to [api] [cloud] add compute instance price to container record
Actions #15

Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)
Actions #16

Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)
Actions #17

Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)
Actions #18

Updated by Tom Clegg almost 2 years ago

18205-container-cost @ f5512fd7f9abe306740af464551938461033a935 -- developer-run-tests: #3262

... retry wb1 developer-run-tests-apps-workbench-integration: #3508

... retry wb1 developer-run-tests-apps-workbench-integration: #3509

(different wb1 failures each time, sigh)

New fields:
  • containers.cost: cost of the container itself
  • containers.subrequests_cost: total cumulative cost of all container requests submitted by this container
  • container_requests.cumulative_cost: total cost of containers attributable to this request (including cumulative cost of subrequests) excluding containers that were reused, i.e., had already completed when this request was committed.

We might want to tweak the conditions for excluding reused container costs (e.g., currently if four concurrent requests all use the same container, the cost will be added to all four of them -- perhaps it should only be added to one, or split among the four). At this point I'm aiming to get the answer right in the most common/obvious cases, e.g., a workflow run that has some brand new containers and some that were reused from previous runs, so we can unblock #19319.

The cost estimates will not be great when using preemptible instances, since we don't yet ask cloud providers for real price data (#19320).

Actions #19

Updated by Tom Clegg almost 2 years ago

  • Status changed from New to In Progress
Actions #20

Updated by Lucas Di Pentima almost 2 years ago

Sorry for the delay, I'll review this tomorrow first thing in the morning.

Actions #21

Updated by Lucas Di Pentima almost 2 years ago

Just one question:

The rest LGTM, thanks.

Actions #22

Updated by Tom Clegg almost 2 years ago

I considered using a fixed-point type, but they have worse performance characteristics (for sort/filter). The "money" type has the additional "locale-sensitive" caveat: "To avoid problems, before restoring a dump into a new database make sure lc_monetary has the same or equivalent value as in the database that was dumped."

I figure since these numbers are inherently imprecise anyway (there are several reasons why they will never agree exactly with real billing figures) we wouldn't really benefit by rounding to a specified number of decimal places at each database interaction.

WDYT?

Actions #23

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-08-17 sprint to 2022-08-31 sprint
Actions #24

Updated by Lucas Di Pentima almost 2 years ago

Sorry for the delay, didn't got the notification on this comment.

I agree with your comments, LGTM.

Actions #25

Updated by Tom Clegg almost 2 years ago

  • Status changed from In Progress to Resolved
Actions #26

Updated by Peter Amstutz over 1 year ago

  • Release set to 47
Actions

Also available in: Atom PDF