Arvados

developer-run-tests: #3460

• keep-balance -chunk-prefix=abc will only balance blocks whose hashes start with abc
• (regardless of chunk mode) while retrieving indexes, keep-balance logs "index progress: received {count} blocks from {keepstore-mount-url}" every 5 minutes
• (regardless of chunk mode) during a balancing operation, keep-balance checks its process memory size once per second, and reports it in a log message if, since the last report, it has grown 40% or 10 minutes have passed

Some questions:

• I think it would be useful to have validations on --chunk-prefix param. If they were skipped because it's an experimental feature, please disregard this comment.
• I don't understand the utility of sleeping one second on services/keep-balance/balance.go line 159, is this something that leaked from testing?
• Is it possible to get mem usage stats from https://pkg.go.dev/runtime#MemStats ? I'm thinking that if we can avoid reading the /proc fs, it's a step forward to be possible to use keep-balance on different OSes.

I think sanity-checking the prefix is a good idea -- the most likely error is passing uppercase hex digits, which would confusingly never match anything. Added.

Oops, the sleep was indeed just for testing. Removed.

I did consider using the Go runtime features to track memory use, but went with procfs instead so that (if there's any difference at all) the stats will match "top" etc.

The portability point is fair, though. Another reason to use the Go runtime is that the operator already has the option to watch "top" if they just want the OS-reported numbers. What's better -- try procfs first and fall back on the Go runtime? Or just use the Go runtime?

Another point: the rss figure might also be interesting, but in the context of tracking/predicting the resources needed to do the job, I thought vsize would be more relevant -- if pages are being swapped out, rss will appear low, which seems misleading. Does this sound right?

19923-keep-balance-chunk @ 1ddc21430ede024aa6841e6c49deafe87788c287 -- developer-run-tests: #3462

Tom Clegg wrote in #note-5:

The portability point is fair, though. Another reason to use the Go runtime is that the operator already has the option to watch "top" if they just want the OS-reported numbers. What's better -- try procfs first and fall back on the Go runtime? Or just use the Go runtime?

Using procfs as a first option is a nice idea, so that there's no difference between what the operator sees in "top" and what's reported in the logs.

Another point: the rss figure might also be interesting, but in the context of tracking/predicting the resources needed to do the job, I thought vsize would be more relevant -- if pages are being swapped out, rss will appear low, which seems misleading. Does this sound right?

If it isn't too much work having both reported, I think it would be useful in cases where keep-balance is being run on a shared node and getting OOMed intermittently because of some other memory hungry process.

The rest LGTM.

• reports HeapInuse, HeapAlloc, and StackInuse from the Go runtime
• reports virt and res from procfs if available
• bases the "increased by 40%" trigger on heapInuse+stackInuse

19923-keep-balance-chunk @ 7d3a6de4ee5afb635e561c75d4f99d1067f030f2 -- developer-run-tests: #3464

LGTM, thanks!