Project

General

Profile

Actions

Feature #19675

closed

Panel that lists configured instance types

Added by Peter Amstutz over 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Workbench2
Story points:
3.0
Release relationship:
Auto

Description

Render the contents of config.InstanceTypes nicely for the benefit of workflow authors.

For RAM and disk, subtract out the overhead that gets added to the resource request in order to show the maximum resource request that can be made in CWL that will match that type, without bumping up to a larger size.

Should also report value of MaxInstances and SupervisorFraction.


Files

Instancetypes.png (144 KB) Instancetypes.png Peter Amstutz, 10/27/2023 04:04 PM

Subtasks 1 (0 open1 closed)

Task #20332: Review 19675-instance-types-panelResolvedStephen Smith12/05/2023Actions

Related issues

Related to Arvados Epics - Idea #16943: WB2 Descriptions, project & collection operationsIn Progress10/01/202206/30/2024Actions
Related to Arvados Epics - Idea #18179: Better spot instance supportIn Progress03/01/202206/30/2024Actions
Is duplicate of Arvados - Feature #12179: Panel showing available instance types on the clusterNewActions
Actions #1

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
  • Subject changed from Panel that lists configured instance types and the largest resource request that will select that type to Panel that lists configured instance types
Actions #2

Updated by Peter Amstutz over 1 year ago

  • Is duplicate of Feature #12179: Panel showing available instance types on the cluster added
Actions #3

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-12-07 Sprint to 2022-12-21 Sprint
Actions #4

Updated by Peter Amstutz over 1 year ago

  • Story points set to 3.0
  • Subject changed from Panel that lists configured instance types to Panel that lists configured instance types
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-12-21 Sprint to 2023-01-18 sprint
Actions #6

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2023-01-18 sprint to 2023-02-01 sprint
Actions #7

Updated by Peter Amstutz over 1 year ago

  • Release set to 59
  • Target version deleted (2023-02-01 sprint)
Actions #8

Updated by Peter Amstutz over 1 year ago

  • Target version set to To be scheduled
Actions #9

Updated by Peter Amstutz about 1 year ago

  • Description updated (diff)
Actions #10

Updated by Peter Amstutz about 1 year ago

  • Related to Idea #16943: WB2 Descriptions, project & collection operations added
Actions #11

Updated by Peter Amstutz about 1 year ago

  • Target version changed from To be scheduled to Development 2023-04-26 sprint
Actions #12

Updated by Peter Amstutz about 1 year ago

  • Assigned To set to Stephen Smith
Actions #13

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-04-26 sprint to Development 2023-05-10 sprint
Actions #14

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-05-10 sprint to Development 2023-05-24 sprint
Actions #15

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-05-24 sprint to Development 2023-06-07
Actions #16

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-06-07 to To be scheduled
Actions #17

Updated by Peter Amstutz 12 months ago

  • Related to Idea #18179: Better spot instance support added
Actions #18

Updated by Peter Amstutz 9 months ago

  • Target version changed from To be scheduled to Development 2023-11-08 sprint
Actions #19

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-10-25 sprint
Actions #20

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Actions #21

Updated by Peter Amstutz 8 months ago

  • File Instance types.png added

Actions #23

Updated by Peter Amstutz 8 months ago

  • File deleted (Instancetypes.png)
Actions #24

Updated by Peter Amstutz 8 months ago

Here's the Go code that calculates the actual RAM request.

So we want to subtract all the overhead from the topline instance RAM to get the maximum value of "RuntimeConstraints.RAM" that the user can provide to get this instance type.

    needRAM := ctr.RuntimeConstraints.RAM + ctr.RuntimeConstraints.KeepCacheRAM
    needRAM += int64(cc.Containers.ReserveExtraRAM)
    if cc.Containers.LocalKeepBlobBuffersPerVCPU > 0 {
        // + 200 MiB for keepstore process + 10% for GOGC=10
        needRAM += 220 << 20
        // + 64 MiB for each blob buffer + 10% for GOGC=10
        needRAM += int64(cc.Containers.LocalKeepBlobBuffersPerVCPU * needVCPUs * (1 << 26) * 11 / 10)
    }
    needRAM = (needRAM * 100) / int64(100-discountConfiguredRAMPercent)
Actions #25

Updated by Brett Smith 8 months ago

Peter Amstutz wrote in #note-24:

Here's the Go code that calculates the actual RAM request.

#20801 is tangentially related. If this ticket happens first, sure, fine, but then if we ever want to do #20801, it should probably grow to include undoing the math in Workbench 2 as well.

Actions #26

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #27

Updated by Peter Amstutz 7 months ago

  • Status changed from New to In Progress
Actions #28

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2023-11-29 sprint to Development 2024-01-03 sprint
Actions #29

Updated by Stephen Smith 7 months ago

Changes at arvados-workbench2|038d67a8c9ebf8a462774e9b870e7f1603c01959 branch 19675-instance-types-panel
Tests developer-tests-workbench2: #1397

  • All agreed upon points are implemented / addressed.
    • Added instance types panel requiring adding a few fields to the arvados config type
    • Add breadcrumbs for instance types panel
    • Add unit tests for instance types panel
    • Add missing breadcrumbs navigation handlers
    • Move process spec 'no value io parameter test' loginas into createContainerRequest then body and use admin user which is the only way I've figured out how to make that test pass on jenkins (it works fine locally for some reason)
  • Anything not implemented (discovered or discussed during work) has a follow-up story.
    • Added #21253 to add MaxInstances and SupervisorFraction to the exported config so that #21256 can add them in the top box
  • Code is tested and passing, both automated and manual, what manual testing was done is described
    • done
  • Documentation has been updated.
    • n/a
  • Behaves appropriately at the intended scale (describe intended scale).
    • intended scale is expected to be on the order of <50 instance types, beyond that the UI might be difficult to use
  • Considered backwards and forwards compatibility issues between client and server.
    • n/a
  • Follows our coding standards and GUI style guidelines.
    • done
Actions #30

Updated by Stephen Smith 7 months ago

I merged the changes into a branch on the main arvados repo at arvados|ee9d1e39b5d469a827be5a719c9c0860914ab2a8 branch 19675-instance-types-panel

It turns out the cypress test fix didn't actually fix the issue, it's still just flakey in probably the same way that Tom was trying to fix, so I was able to get it to pass without my fix by rerunning: developer-run-tests-services-workbench2: #84

Actions #31

Updated by Peter Amstutz 6 months ago

Reviewing 19675-instance-types-panel @ ee9d1e39b5d469a827be5a719c9c0860914ab2a8

  • These don't seem to be in an particular order. Can we sort them by price from lowest to highest?
  • RAM should be capitalized in "Max ram request"
  • Even though LocalKeepBlobBuffersPerVCPU isn't exported, I think we should calculate the maximum request assuming it is the default value. Also we should include the 5% discount (also assuming the default behavior for now). In other words, it should account for all the stuff in note-24, and just assume defaults where the configured values are not available.
    • defaults are LocalKeepBlobBuffersPerVCPU: 1 and discountConfiguredRAMPercent: 5
    • Since this is user-facing to help users set correct resource requests, it is going to be better if the values that are displayed under-shoot the maximum than show a maximum that is wrong and frustrates users because they don't get the instance type they intended. In the future, if/when we rework instance selection, we can update the formulas (or have the values provided from somewhere else).
  • For CWL, the unit of memory is MiB. Let's display RAM and disk in mebibytes along with whatever units are more human-readable? e.g.
    Max disk request: 16252928 MiB (7.3 TiB)
    Max ram request: 15872 MiB (15.5 GiB)
  • I realize the design sketch was a little weird, the order of the fields should be:
    • provider type
    • price
    • cores
    • RAM
    • disk
    • preemptible
    • GPU stuff
  • It is currently a bunch of little cards on one big card (white on white). Can we get rid of the outer container card?
  • Let's add it to the left panel, underneath "All Processes". In general I think we want to move towards phasing out the "user" menu except for user account-related stuff.
  • Instead of the stack-of-servers icon, let's use the same stylized computer chip that is used for the "Resources" card of the process view.
    • Speaking of the "Resources" card, I feel like they should be formatted the same way (with the same labels, as much as possible, to emphasize the connection between the numbers), on the other hand having the field name on the top instead of to the left might needlessly waste vertical space, so I'm not sure.
  • We should have some text at the top explaining what this page is and how to use it. E.g. "These are the cloud compute instance types configured for this cluster. The core count and maximum RAM request correspond to the greatest values you can put in the CWL Workflow ResourceRequest `minCores` and `minRAM` and still be scheduled on that instance type."
Actions #32

Updated by Stephen Smith 6 months ago

Changes at arvados|14a2e78ff5cd055216a9f6edc40dfb3f79122fa8 branch 19675-instance-types-panel
Tests developer-run-tests-services-workbench2: #98

  • Instance types sorted by increasing cost, with an alphabetical sort of the ProviderType as a tiebreaker
  • Fixed capitalization and field order
  • Added keep buffer and discount ram percent accounting
  • Added "CWL" size formatting and display MiB next to convenient units
  • Removed container card
  • Moved panel link to left panel
  • Switched to resources icon
  • Switched to DetailsAttribute component to better match resources panel
  • Added info box to the top with the suggested text
Actions #33

Updated by Peter Amstutz 6 months ago

Stephen Smith wrote in #note-32:

Changes at arvados|14a2e78ff5cd055216a9f6edc40dfb3f79122fa8 branch 19675-instance-types-panel
Tests developer-run-tests-services-workbench2: #98

  • Instance types sorted by increasing cost, with an alphabetical sort of the ProviderType as a tiebreaker
  • Fixed capitalization and field order
  • Added keep buffer and discount ram percent accounting
  • Added "CWL" size formatting and display MiB next to convenient units
  • Removed container card
  • Moved panel link to left panel
  • Switched to resources icon
  • Switched to DetailsAttribute component to better match resources panel
  • Added info box to the top with the suggested text

This looks good, just one thing I noticed:

  • The icon is missing from the collapsed left side panel. It appears correctly in the expanded left panel.

Rest LGTM.

Actions #34

Updated by Stephen Smith 6 months ago

  • Status changed from In Progress to Resolved
Actions #35

Updated by Peter Amstutz 5 months ago

  • Release set to 69
Actions

Also available in: Atom PDF