Feature #20756
openSupport crunchstat tracking and memory limits with singularity
0%
Description
Singularity has capability to put the container in a new cgroup and set resource usage limits. Even without applying any limits, this also enables resource usage tracking by crunchstat.
https://docs.sylabs.io/guides/3.0/user-guide/cgroups.html
The docs say "the --apply-cgroups
option can only be used with root privileges" but these tests worked as a non-root user:
$ singularity version 3.10.4-dirty $ singularity exec --apply-cgroups /dev/null docker://debian:12 sleep 600 & [1] 60133 $ pstree -up | grep sleep | | `-starter-suid(60133)-+-sleep(60151) $ cat /proc/60133/cgroup 0::/user.slice/user-1000.slice/session-5424.scope $ cat /proc/60151/cgroup 0::/user.slice/user-1000.slice/user@1000.service/user.slice/singularity-60151.scope $ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/user.slice/singularity-60151.scope/memory.current 2465792
$ singularity exec --apply-cgroups <(printf '[memory]\n limit = 5000000\n') docker://debian:12 echo ok ok $ singularity exec --apply-cgroups <(printf '[memory]\n limit = 5000\n') docker://debian:12 echo ok Killed
As of #17244 crunch-run does not correctly identify the pid of a process inside the container when telling crunchstat which process/cgroup to monitor (it returns the pid of the singularity executor wrapper instead). This will also need to be fixed in order for crunchstat to work correctly.
Related issues
Updated by Tom Clegg 5 months ago
- Blocked by Bug #17244: Make sure cgroupsV2 works with Arvados added
Updated by Brett Smith 5 months ago
Tom Clegg wrote:
The docs say "the
--apply-cgroups
option can only be used with root privileges" but these tests worked as a non-root user:
- It's possible that line was written before user namespaces were widely available/enabled, and has become obsolete since. The timeline kinda works: Singularity 3.0.0 was released October 2018, and Debian got user namespaces in 11, released August 2021.
- But also, if you're going through
starter-suid
, don't you have root privileges at some level?
Updated by Peter Amstutz 4 months ago
- Target version changed from To be groomed to Development 2023-08-30
- Assigned To set to Tom Clegg
Updated by Peter Amstutz 3 months ago
- Target version changed from Development 2023-08-30 to Development 2023-09-13 sprint
Updated by Peter Amstutz 3 months ago
- Target version changed from Development 2023-09-13 sprint to Development 2023-09-27 sprint
Updated by Peter Amstutz 3 months ago
- Target version changed from Development 2023-09-27 sprint to To be scheduled