Project

General

Profile

Actions

Feature #20756

open

Support crunchstat tracking and memory limits with singularity

Added by Tom Clegg 11 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Docker
Target version:
Story points:
2.0

Description

Singularity has capability to put the container in a new cgroup and set resource usage limits. Even without applying any limits, this also enables resource usage tracking by crunchstat.

https://docs.sylabs.io/guides/3.0/user-guide/cgroups.html

The docs say "the --apply-cgroups option can only be used with root privileges" but these tests worked as a non-root user:

$ singularity version
3.10.4-dirty
$ singularity exec --apply-cgroups /dev/null docker://debian:12 sleep 600 &
[1] 60133
$ pstree -up | grep sleep
           |                     |             `-starter-suid(60133)-+-sleep(60151)
$ cat /proc/60133/cgroup
0::/user.slice/user-1000.slice/session-5424.scope
$ cat /proc/60151/cgroup
0::/user.slice/user-1000.slice/user@1000.service/user.slice/singularity-60151.scope
$ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/user.slice/singularity-60151.scope/memory.current 
2465792
$ singularity exec --apply-cgroups <(printf '[memory]\n limit = 5000000\n') docker://debian:12 echo ok
ok
$ singularity exec --apply-cgroups <(printf '[memory]\n limit = 5000\n') docker://debian:12 echo ok
Killed

As of #17244 crunch-run does not correctly identify the pid of a process inside the container when telling crunchstat which process/cgroup to monitor (it returns the pid of the singularity executor wrapper instead). This will also need to be fixed in order for crunchstat to work correctly.


Subtasks 1 (1 open0 closed)

Task #20870: ReviewNewLucas Di PentimaActions

Related issues

Blocked by Arvados - Bug #17244: Make sure cgroupsV2 works with ArvadosResolvedTom Clegg07/18/2023Actions
Actions #1

Updated by Tom Clegg 11 months ago

  • Target version set to Future
  • Category set to Docker
  • Tracker changed from Bug to Feature
Actions #2

Updated by Tom Clegg 11 months ago

  • Blocked by Bug #17244: Make sure cgroupsV2 works with Arvados added
Actions #3

Updated by Brett Smith 11 months ago

Tom Clegg wrote:

The docs say "the --apply-cgroups option can only be used with root privileges" but these tests worked as a non-root user:

  1. It's possible that line was written before user namespaces were widely available/enabled, and has become obsolete since. The timeline kinda works: Singularity 3.0.0 was released October 2018, and Debian got user namespaces in 11, released August 2021.
  2. But also, if you're going through starter-suid, don't you have root privileges at some level?
Actions #4

Updated by Peter Amstutz 10 months ago

  • Target version changed from Future to Development 2023-08-30
  • Assigned To set to Tom Clegg
Actions #5

Updated by Peter Amstutz 10 months ago

  • Target version changed from Development 2023-08-30 to Development 2023-09-13 sprint
Actions #6

Updated by Peter Amstutz 10 months ago

  • Target version changed from Development 2023-09-13 sprint to Development 2023-09-27 sprint
Actions #7

Updated by Peter Amstutz 9 months ago

  • Story points set to 2.0
Actions #8

Updated by Peter Amstutz 9 months ago

  • Target version changed from Development 2023-09-27 sprint to To be scheduled
Actions #9

Updated by Peter Amstutz 4 months ago

  • Target version changed from To be scheduled to Future
Actions

Also available in: Atom PDF