Project

General

Profile

Actions

Feature #22157

open

Record resource usage high water marks in the container record

Added by Peter Amstutz 3 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Crunch
Target version:
Story points:
-

Description

It would be very helpful to have the high water marks for CPU and RAM usage of a container recorded on the container record. There's two main uses I can think of:

  • Detecting likely OOM conditions. Right now, arvados-cwl-runner parses crunch-run.log to get this information and put it as a runtime status warning. This means if arvados-cwl-runner itself goes OOM, it doesn't get flagged to the user.
  • Detecting under-utilization. If CPU or RAM usage is < 50% for the whole run, it's a candidate for reducing the resource request. However, before changing the workflow, it is important to know if the step is consistently under-utilizing its resources, or if it's highly variable. Having this in the database makes the information way more accessible than having to parse logs.
Actions #1

Updated by Peter Amstutz 3 months ago

  • Description updated (diff)
Actions #2

Updated by Peter Amstutz 3 months ago

  • Target version changed from Future to Development 2024-11-06 sprint
Actions #3

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-11-06 sprint to Future
Actions

Also available in: Atom PDF