Idea #9653
Updated by Tom Clegg over 6 years ago
It's hard to predict the right setting for @-max-buffers@.
* The sysadmin must coordinate configuration files with host characteristics: e.g., when upgrading to a bigger node, the sysadmin must also remember to change max-buffers at the same time.
* The sysadmin must account for (and/or override) Go's garbage collection threshold, likely 100% (default) or 10% (our recommendation).
* The sysadmin must predict how much memory keepstore will use for purposes other than {buffers + 10% garbage collection overhead}, and reduce max-buffers accordingly.
* The sysadmin must predict how much memory will be used by other processes running on the same node (e.g., cloud/monitoring/orchestration agents).
In addition to a static limit on the number of buffers, we could configure a maximum keepstore memory size, checking the and check actual process memory usage (not just bytes-in-buffers) before allocating a new buffer. This isn't perfect (it wouldn't be a hard limit) doesn't limit allocations for anything other than buffers) but it would be strictly safer and easier than the current behavior. behavior, which _never_ accounts for other allocations.
Memory use could be limited by providing one or more of these configs:
<pre>
# Maximum memory use, as a number of MiB
MaxMemoryMiB: 6000
# Maximum memory use, as a percentage of total system RAM
MaxMemoryPercent: 80
</pre>
Suggested default:
<pre>
MaxMemoryPercent: 80
</pre>
To address the last difficulty (predicting memory use by other processes) we could also offer a MinMemoryFree or MinMemoryAvailable config, which avoids allocating more buffers when Linux reports a low MemFree or MemAvailable.
More than one limit can be configured at once. Keepstore should allocate new buffers only when none of the configured limits will be exceeded.