Idea #14996
open
Log crunch-run resource usage statistics
Added by Tom Morris over 5 years ago.
Updated 8 months ago.
Release relationship:
Auto
Description
Currently we have crunchstat.txt for container stats and hoststat.txt for node level stats, but we have no way of recording the time and resources for file upload by crunch-run after the container exits. This would be very useful since, for I/O intensive jobs like bcl2fastq, this can represent as much time as the compute phase of the job.
In order to collect fine-grained metrics, one option is for crunch-run to have a bootstrap mode that creates a cgroup and runs crunch-run and arv-mount inside it.
Go cgroup module should make this easier https://godoc.org/github.com/containerd/cgroups. Might be worth trying to have crunch-run move itself to a cgroup in bootstrap mode. If that doesn't work, we'd have to start another process.
Once crunch-run and arv-mount are in their own cgroup, the arvados crunchstat go module can automatically interpret stats.
- Target version changed from To Be Groomed to Arvados Future Sprints
- Target version deleted (
Arvados Future Sprints)
- Target version set to Future
Also available in: Atom
PDF