Idea #14996
open
Log crunch-run resource usage statistics
Description
Currently we have crunchstat.txt for container stats and hoststat.txt for node level stats, but we have no way of recording the time and resources for file upload by crunch-run after the container exits. This would be very useful since, for I/O intensive jobs like bcl2fastq, this can represent as much time as the compute phase of the job.
Updated by Peter Amstutz over 5 years ago
In order to collect fine-grained metrics, one option is for crunch-run to have a bootstrap mode that creates a cgroup and runs crunch-run and arv-mount inside it.
Updated by Eric Biagiotti over 5 years ago
- Story points set to 2.0
Go cgroup module should make this easier https://godoc.org/github.com/containerd/cgroups. Might be worth trying to have crunch-run move itself to a cgroup in bootstrap mode. If that doesn't work, we'd have to start another process.
Once crunch-run and arv-mount are in their own cgroup, the arvados crunchstat go module can automatically interpret stats.
Updated by Tom Morris over 5 years ago
- Target version changed from To Be Groomed to Arvados Future Sprints
Updated by Peter Amstutz over 3 years ago
- Target version deleted (
Arvados Future Sprints)