Log crunch-run resource usage statistics
Currently we have crunchstat.txt for container stats and hoststat.txt for node level stats, but we have no way of recording the time and resources for file upload by crunch-run after the container exits. This would be very useful since, for I/O intensive jobs like bcl2fastq, this can represent as much time as the compute phase of the job.
#2 Updated by Eric Biagiotti 2 months ago
- Story points set to 2.0
Go cgroup module should make this easier https://godoc.org/github.com/containerd/cgroups. Might be worth trying to have crunch-run move itself to a cgroup in bootstrap mode. If that doesn't work, we'd have to start another process.
Once crunch-run and arv-mount are in their own cgroup, the arvados crunchstat go module can automatically interpret stats.