Project

General

Profile

Actions

Feature #20756

open

Support crunchstat tracking and memory limits with singularity

Added by Tom Clegg 7 months ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Docker
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
2.0

Description

Singularity has capability to put the container in a new cgroup and set resource usage limits. Even without applying any limits, this also enables resource usage tracking by crunchstat.

https://docs.sylabs.io/guides/3.0/user-guide/cgroups.html

The docs say "the --apply-cgroups option can only be used with root privileges" but these tests worked as a non-root user:

$ singularity version
3.10.4-dirty
$ singularity exec --apply-cgroups /dev/null docker://debian:12 sleep 600 &
[1] 60133
$ pstree -up | grep sleep
           |                     |             `-starter-suid(60133)-+-sleep(60151)
$ cat /proc/60133/cgroup
0::/user.slice/user-1000.slice/session-5424.scope
$ cat /proc/60151/cgroup
0::/user.slice/user-1000.slice/user@1000.service/user.slice/singularity-60151.scope
$ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/user.slice/singularity-60151.scope/memory.current 
2465792
$ singularity exec --apply-cgroups <(printf '[memory]\n limit = 5000000\n') docker://debian:12 echo ok
ok
$ singularity exec --apply-cgroups <(printf '[memory]\n limit = 5000\n') docker://debian:12 echo ok
Killed

As of #17244 crunch-run does not correctly identify the pid of a process inside the container when telling crunchstat which process/cgroup to monitor (it returns the pid of the singularity executor wrapper instead). This will also need to be fixed in order for crunchstat to work correctly.


Subtasks 1 (1 open0 closed)

Task #20870: ReviewNewLucas Di Pentima

Actions

Related issues

Blocked by Arvados - Bug #17244: Make sure cgroupsV2 works with ArvadosResolvedTom Clegg07/18/2023

Actions
Actions

Also available in: Atom PDF