Feature #18677

Container runtime metrics API

Added by Tom Clegg 4 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

New API endpoint: arvados/v1/containers/{uuid}/metrics
  • handled in lib/controller/localdb (not RailsAPI, although RailsAPI might add a route for the sake of getting it published in the discovery doc)
  • use collection filesystem to read crunchstat.txt from log collection
  • rates like net.*.tx and cpu.user are given as per-second values
  • mem.rss is given as absolute/maximum value during preceding interval
  • seconds = time since container start / first data point
  • each series has the same number of elements
  • series other than "seconds" may have null elements (meaning data point not available)
Limitations in this initial implementation phase:
  • only accesses data points that have been saved to the container's log collection (typically the last half hour will not be available)
  • no downsample/crop/truncate options -- the response might be large for a long-running container
  • no server-side cache
  • only provides seconds, mem.rss, net:eth0.tx, net:eth0.rx, net:keep0.tx, net:keep0.rx, cpu.user, cpu.sys -- other metrics tracked in crunchstat.txt and arv-mount.txt are ignored

Example API response:

{
  "samples": {
    "seconds": [
      0,
      10.01,
      19.99,
      21.12,
      30.03
    ],
    "mem.rss": [
      100000,
      1000000,
      1200000,
      1300000,
      1250000
    ],
    "net:eth0.tx": [
      2591244,
      221646,
      76284,
      36588,
      11578
    ],
    "net:eth0.rx": [
      801655428,
      161832266,
      58664743,
      33127923,
      14192042
    ],
    "cpu.user": [
      61.984,
      66.346,
      63.344,
      75.972,
      50.729
    ],
    "cpu.sys": [
      14.099,
      25.574,
      30.229,
      17.513,
      42.085
    ]
  }
}


Related issues

Related to Arvados - Story #17472: Architecture for displaying container metrics in workbenchIn Progress

Related to Arvados Epics - Story #16517: Workflow runtime/cost visibility and forecastingIn Progress10/01/202204/30/2023

History

#1 Updated by Tom Clegg 4 months ago

  • Related to Story #17472: Architecture for displaying container metrics in workbench added

#2 Updated by Tom Clegg 4 months ago

  • Related to Story #16517: Workflow runtime/cost visibility and forecasting added

Also available in: Atom PDF