Project

General

Profile

Actions

Feature #20983

open

Tool to automatically populate Arvados InstanceTypes

Added by Peter Amstutz 9 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
Story points:
3.0

Description

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypeOfferings.html

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-ppslong.html

Write a tool that gets all the instance type info from AWS (RAM/CPU spec, included disk, reserved price), filters the list with the options listed below, and print out (or update) the InstanceTypes section of the Arvados config file.

It looks like the price list has the relevant info, e.g.

In the "products" section:

"sku" : "QUMEF4UK3NPT4MN3"
"productFamily" : "Compute Instance",
"vcpu" : "4"
"memory" : "7.5 GiB"
"storage" : "2 x 40 SSD" (I don't know how multiple instance disks are presented to the operating system or if we support this in our current compute image)
"marketoption" : "OnDemand",

In the "terms" "OnDemand" section:

"sku" : "QUMEF4UK3NPT4MN3"
"unit" : "Hrs",
"pricePerUnit" : {
"USD" : "0.3760000000"
},

Include a flag to indicate if ebs autoscale is in use. If EBS autoscale is enabled the disk section should specify 5 TiB of included storage.

If not using EBS autoscale, flag to set default # of GiB of attached storage for "EBS Only" volumes.

Include flag to specify one or more availability zones and get the instance types for those availability zones.

Include flags to specify minimum cores, minimum RAM, and maximum price.

This tool should preferably be written in Go and added as a sub-command of arvados-server.

Example AWS command line for getting instance types in an availability zone (apparently there's other ways of querying for this but they are incredibly slow)

aws ec2 describe-instance-type-offerings --filters "Name=location,Values=us-east-1a" --location-type availability-zone --region us-east-1

If we do end up using DescribeInstanceTypes, here's the relevant fields:

instanceStorageInfo, memoryInfo, vCpuInfo

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypes.html


Subtasks 1 (1 open0 closed)

Task #21008: ReviewNewPeter AmstutzActions

Related issues

Related to Arvados Epics - Idea #20599: Scaling to 1000s of concurrent containersResolved06/01/202303/31/2024Actions
Related to Arvados - Feature #20978: Support multiple candidate instance types to assign containersResolvedTom Clegg10/31/2023Actions
Actions #1

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz 9 months ago

  • Related to Idea #20599: Scaling to 1000s of concurrent containers added
Actions #3

Updated by Peter Amstutz 9 months ago

  • Story points set to 3.0
  • Target version changed from Future to To be scheduled
  • Description updated (diff)
Actions #4

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz 9 months ago

  • Subject changed from Use DescribeInstanceTypes to automatically populate Arvados InstanceTypes to Tool to automatically populate Arvados InstanceTypes
Actions #6

Updated by Peter Amstutz 9 months ago

  • Target version changed from To be scheduled to Development 2023-10-11 sprint
Actions #7

Updated by Brett Smith 9 months ago

I don't understand why the tool is specced to filter by regexp on instance name. It seems to me like users are most likely to be able to want to say: I want to use these classes (t5/m5/c5/etc.) with these minimum specs (e.g., at least 2 cores, at least 1GiB RAM, etc.) and up to this maximum hourly cost. Can we give them that? It is a little more work but doesn't seem outrageous.

Actions #8

Updated by Peter Amstutz 9 months ago

I was assuming that you would go in and manually delete the things you don't want, but if there's whole classes of instance types you don't want, a regex is a straightforward to filter them out.

Unfortunately the API itself only supports exact matches and "in set", it doesn't support greater-than or less-than comparisons.

So we can add a few client side filters but I don't want to over-complicate it.

If we want an arbitrary query language to filter we could also pipe through jq.

Actions #9

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #10

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #11

Updated by Peter Amstutz 9 months ago

  • Related to Feature #20978: Support multiple candidate instance types to assign containers added
Actions #12

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #13

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #14

Updated by Peter Amstutz 9 months ago

  • Assigned To set to Brett Smith
Actions #15

Updated by Tom Clegg 9 months ago

  • Description updated (diff)
Actions #16

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #17

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-10-11 sprint to Development 2023-10-25 sprint
Actions #18

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Actions #19

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #20

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-11-29 sprint to Development 2024-01-03 sprint
Actions #21

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2024-01-03 sprint to Development 2024-01-17 sprint
Actions #22

Updated by Peter Amstutz 6 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-03-27 sprint
Actions #23

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-03-27 sprint to To be scheduled
Actions #24

Updated by Peter Amstutz 4 months ago

  • Target version changed from To be scheduled to Future
Actions

Also available in: Atom PDF