Idea #9653
open[keepstore] Limit entire process to configured amount or % of system memory (replace -max-buffers=X)
Description
-max-buffers
.
- The sysadmin must coordinate configuration files with host characteristics: e.g., when upgrading to a bigger node, the sysadmin must also remember to change max-buffers at the same time.
- The sysadmin must account for (and/or override) Go's garbage collection threshold, likely 100% (default) or 10% (our recommendation).
- The sysadmin must predict how much memory keepstore will use for purposes other than {buffers + 10% garbage collection overhead}, and reduce max-buffers accordingly.
- The sysadmin must predict how much memory will be used by other processes running on the same node (e.g., cloud/monitoring/orchestration agents).
In addition to a static limit on the number of buffers, we could configure a maximum keepstore memory size, checking the actual process memory usage (not just bytes-in-buffers) before allocating a new buffer. This isn't perfect (it wouldn't be a hard limit) but it would be strictly safer and easier than the current behavior.
Memory use could be limited by providing one or more of these configs:
# Maximum memory use, as a number of MiB MaxMemoryMiB: 6000 # Maximum memory use, as a percentage of total system RAM MaxMemoryPercent: 80
Suggested default:
MaxMemoryPercent: 80
To address the last difficulty (predicting memory use by other processes) we could also offer a MinMemoryFree or MinMemoryAvailable config, which avoids allocating more buffers when Linux reports a low MemFree or MemAvailable.
More than one limit can be configured at once. Keepstore should allocate new buffers only when none of the configured limits will be exceeded.