Story #9653
[keepstore] Limit entire process to configured amount or % of system memory (replace -max-buffers=X)
0%
Description
-max-buffers
.
- The sysadmin must coordinate configuration files with host characteristics: e.g., when upgrading to a bigger node, the sysadmin must also remember to change max-buffers at the same time.
- The sysadmin must account for (and/or override) Go's garbage collection threshold, likely 100% (default) or 10% (our recommendation).
- The sysadmin must predict how much memory keepstore will use for purposes other than {buffers + 10% garbage collection overhead}, and reduce max-buffers accordingly.
- The sysadmin must predict how much memory will be used by other processes running on the same node (e.g., cloud/monitoring/orchestration agents).
In addition to a static limit on the number of buffers, we could configure a maximum keepstore memory size, checking the actual process memory usage (not just bytes-in-buffers) before allocating a new buffer. This isn't perfect (it wouldn't be a hard limit) but it would be strictly safer and easier than the current behavior.
Memory use could be limited by providing one or more of these configs:
# Maximum memory use, as a number of MiB MaxMemoryMiB: 6000 # Maximum memory use, as a percentage of total system RAM MaxMemoryPercent: 80
Suggested default:
MaxMemoryPercent: 80
To address the last difficulty (predicting memory use by other processes) we could also offer a MinMemoryFree or MinMemoryAvailable config, which avoids allocating more buffers when Linux reports a low MemFree or MemAvailable.
More than one limit can be configured at once. Keepstore should allocate new buffers only when none of the configured limits will be exceeded.
History
#1
Updated by Tom Clegg about 3 years ago
- Subject changed from [Keep] Improve default for keepstore -max-buffers to [keepstore] Limit entire process to configured amount or % of system memory (replace -max-buffers=X)
- Description updated (diff)
#2
Updated by Tom Clegg about 3 years ago
- Description updated (diff)
#3
Updated by Tom Morris about 3 years ago
- Target version set to To Be Groomed
#4
Updated by Tom Morris about 3 years ago
I'd suggest that we bias this to work reliably at the expense of not using every last nibble of RAM.
Perhaps a default scheme along the lines of this: https://www.datacore.com/SSV-Webhelp/Changing_Cache_Size.htm
with values tweaked appropriately.
o RAM less than or equal to 8 GB; the cache will use 60%. (8 GB is the minimum requirement)
o RAM less than or equal to 128 GB; the cache will use 65%.
o RAM greater than 128 GB; the cache will use 75%.