Arvados

Currently, crunch-dispatch-slurm sets the memory limit in the sbatch command by adding the RAM runtime constraint and the KeepCacheRAM runtime constraint: https://github.com/curoverse/arvados/blob/master/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go#L209

When running on SLURM with cgroup memory limits enabled, this can result in all containers that actually use close to the memory limit (such as pretty much anything involving the JVM inside the container) failing.

I believe this is because, in addition to the KeepCacheRAM, both crunch-run and arv-mount also use additional memory for their internal data structures such as the keep manifest caches, fuse inode mappings, etc. Ultimately, there should probably be a story to properly sort out how to set the memory limit to something appropriate and then stick within that limit. However, for now it would be helpful to just have a simple config option to add a specified constant amount of memory to every SLURM request as a reservation for what crunch-run and arv-mount use outside of the container.

I plan to implement this and submit a pull request shortly.