Arvados

It's hard to predict the right setting for -max-buffers.

• The sysadmin must coordinate configuration files with host characteristics: e.g., when upgrading to a bigger node, the sysadmin must also remember to change max-buffers at the same time.
• The sysadmin must account for (and/or override) Go's garbage collection threshold, likely 100% (default) or 10% (our recommendation).
• The sysadmin must predict how much memory keepstore will use for purposes other than {buffers + 10% garbage collection overhead}, and reduce max-buffers accordingly.
• The sysadmin must predict how much memory will be used by other processes running on the same node (e.g., cloud/monitoring/orchestration agents).

In addition to a static limit on the number of buffers, we could configure a maximum keepstore memory size, checking the actual process memory usage (not just bytes-in-buffers) before allocating a new buffer. This isn't perfect (it wouldn't be a hard limit) but it would be strictly safer and easier than the current behavior.

Memory use could be limited by providing one or more of these configs:

# Maximum memory use, as a number of MiB
MaxMemoryMiB: 6000

# Maximum memory use, as a percentage of total system RAM
MaxMemoryPercent: 80

Suggested default:

MaxMemoryPercent: 80

To address the last difficulty (predicting memory use by other processes) we could also offer a MinMemoryFree or MinMemoryAvailable config, which avoids allocating more buffers when Linux reports a low MemFree or MemAvailable.

More than one limit can be configured at once. Keepstore should allocate new buffers only when none of the configured limits will be exceeded.