[Crunch] Contain all job/task containers in a memory-limiting cgroup
Even with #5425, it could improve worker node stability -- and allow some memory oversubscription for task processes -- to have all docker containers run in one memory cgroup with an RSS limit (and run in their own cgroups, of course). The docker daemon itself could be in this outer group too. Presumably the outer cgroup doesn't need to be a full-blown docker container, though -- maybe something like this:
cgcreate -g memory:docker cgset -r memory.limit_in_bytes=4G docker cgexec -g memory:docker docker.io --daemon
I tried this, and it affected the docker daemon itself, but not the containers it created. Perhaps there is a way to make docker inherit memory limits from the outer container? My impression is that cgroups are hierarchical, which should make this sort of thing possible.