Project

General

Profile

Actions

Feature #20983

open

Tool to automatically populate Arvados InstanceTypes

Added by Peter Amstutz about 1 year ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Crunch
Target version:
Story points:
3.0

Description

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypeOfferings.html

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-ppslong.html

Write a tool that gets all the instance type info from AWS (RAM/CPU spec, included disk, reserved price), filters the list with the options listed below, and print out (or update) the InstanceTypes section of the Arvados config file.

It looks like the price list has the relevant info, e.g.

In the "products" section:

"sku" : "QUMEF4UK3NPT4MN3"
"productFamily" : "Compute Instance",
"vcpu" : "4"
"memory" : "7.5 GiB"
"storage" : "2 x 40 SSD" (I don't know how multiple instance disks are presented to the operating system or if we support this in our current compute image)
"marketoption" : "OnDemand",

In the "terms" "OnDemand" section:

"sku" : "QUMEF4UK3NPT4MN3"
"unit" : "Hrs",
"pricePerUnit" : {
"USD" : "0.3760000000"
},

Include a flag to indicate if ebs autoscale is in use. If EBS autoscale is enabled the disk section should specify 5 TiB of included storage.

If not using EBS autoscale, flag to set default # of GiB of attached storage for "EBS Only" volumes.

Include flag to specify one or more availability zones and get the instance types for those availability zones.

Include flags to specify minimum cores, minimum RAM, and maximum price.

This tool should preferably be written in Go and added as a sub-command of arvados-server.

Example AWS command line for getting instance types in an availability zone (apparently there's other ways of querying for this but they are incredibly slow)

aws ec2 describe-instance-type-offerings --filters "Name=location,Values=us-east-1a" --location-type availability-zone --region us-east-1

If we do end up using DescribeInstanceTypes, here's the relevant fields:

instanceStorageInfo, memoryInfo, vCpuInfo

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceTypes.html


Subtasks 1 (1 open0 closed)

Task #21008: ReviewNewPeter AmstutzActions

Related issues

Related to Arvados Epics - Idea #20599: Scaling to 1000s of concurrent containersResolved06/01/202303/31/2024Actions
Related to Arvados - Feature #20978: Support multiple candidate instance types to assign containersResolvedTom Clegg10/31/2023Actions
Actions

Also available in: Atom PDF