Story #14996

Log crunch-run resource usage statistics

Added by Tom Morris 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
2.0

Description

Currently we have crunchstat.txt for container stats and hoststat.txt for node level stats, but we have no way of recording the time and resources for file upload by crunch-run after the container exits. This would be very useful since, for I/O intensive jobs like bcl2fastq, this can represent as much time as the compute phase of the job.

History

#1 Updated by Peter Amstutz 2 months ago

In order to collect fine-grained metrics, one option is for crunch-run to have a bootstrap mode that creates a cgroup and runs crunch-run and arv-mount inside it.

#2 Updated by Eric Biagiotti 2 months ago

  • Story points set to 2.0

Go cgroup module should make this easier https://godoc.org/github.com/containerd/cgroups. Might be worth trying to have crunch-run move itself to a cgroup in bootstrap mode. If that doesn't work, we'd have to start another process.

Once crunch-run and arv-mount are in their own cgroup, the arvados crunchstat go module can automatically interpret stats.

#3 Updated by Tom Morris about 2 months ago

  • Target version changed from To Be Groomed to Arvados Future Sprints

Also available in: Atom PDF