Project

General

Profile

Actions

Feature #18677

open

Container runtime metrics API

Added by Tom Clegg over 2 years ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

New API endpoint: arvados/v1/containers/{uuid}/metrics
  • handled in lib/controller/localdb (not RailsAPI, although RailsAPI might add a route for the sake of getting it published in the discovery doc)
  • use collection filesystem to read crunchstat.txt from log collection
  • rates like net.*.tx and cpu.user are given as per-second values
  • mem.rss is given as absolute/maximum value during preceding interval
  • seconds = time since container start / first data point
  • each series has the same number of elements
  • series other than "seconds" may have null elements (meaning data point not available)
Limitations in this initial implementation phase:
  • only accesses data points that have been saved to the container's log collection (typically the last half hour will not be available)
  • no downsample/crop/truncate options -- the response might be large for a long-running container
  • no server-side cache
  • only provides seconds, mem.rss, net:eth0.tx, net:eth0.rx, net:keep0.tx, net:keep0.rx, cpu.user, cpu.sys -- other metrics tracked in crunchstat.txt and arv-mount.txt are ignored

Example API response:

{
  "samples": {
    "seconds": [
      0,
      10.01,
      19.99,
      21.12,
      30.03
    ],
    "mem.rss": [
      100000,
      1000000,
      1200000,
      1300000,
      1250000
    ],
    "net:eth0.tx": [
      2591244,
      221646,
      76284,
      36588,
      11578
    ],
    "net:eth0.rx": [
      801655428,
      161832266,
      58664743,
      33127923,
      14192042
    ],
    "cpu.user": [
      61.984,
      66.346,
      63.344,
      75.972,
      50.729
    ],
    "cpu.sys": [
      14.099,
      25.574,
      30.229,
      17.513,
      42.085
    ]
  }
}


Related issues

Related to Arvados - Idea #17472: Architecture for displaying container metrics in workbenchIn ProgressTom CleggActions
Related to Arvados Epics - Idea #16517: Workflow resource usage, runtime, and cost visibility and forecastingIn Progress12/31/202307/30/2024Actions
Actions #1

Updated by Tom Clegg over 2 years ago

  • Related to Idea #17472: Architecture for displaying container metrics in workbench added
Actions #2

Updated by Tom Clegg over 2 years ago

  • Related to Idea #16517: Workflow resource usage, runtime, and cost visibility and forecasting added
Actions #3

Updated by Peter Amstutz over 1 year ago

  • Release set to 60
Actions #4

Updated by Peter Amstutz 7 months ago

  • Target version set to Future
Actions

Also available in: Atom PDF