Project

General

Profile

Actions

Story #19923

closed

keep-balance experimental single-chunk mode

Added by Tom Clegg 16 days ago. Updated about 22 hours ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
01/27/2023
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
2.0

Description

Background: These features will enable us to collect data about the impact of sharding on keep-balance time and memory requirements. Our expectation is that sharding will ultimately enable keep-balance to scale by a few more orders of magnitude, and decrease its resource consumption for all cluster sizes.

Features:
  • If a hexadecimal prefix is specified on the command line (e.g., "f" or "0f"), keep-balance only balances blocks with that prefix.
    • This only makes sense as a data-gathering experiment using the -once flag.
    • Statistics will be misleading, because they will still have labels like "total" but only report a small percentage of the blocks.
    • At this stage we will not support an option to automatically run all shards sequentially or concurrently.
  • Improve progress reporting while reading indexes from keepstore servers: e.g., every 5 minutes log "{volume}: received {N} blocks so far"
  • Periodically report keep-balance's own memory usage

Subtasks 1 (0 open1 closed)

Task #19943: Review 19923-keep-balance-chunkResolvedTom Clegg01/27/2023

Actions
Actions #1

Updated by Peter Amstutz 10 days ago

  • Target version changed from To be scheduled to 2023-02-01 sprint
Actions #2

Updated by Tom Clegg 4 days ago

  • Status changed from New to In Progress
Actions #3

Updated by Tom Clegg 1 day ago

19923-keep-balance-chunk @ 7fd14a123d0099c5dcfb0646125afa0b643bd6d2 -- developer-run-tests: #3460
  • keep-balance -chunk-prefix=abc will only balance blocks whose hashes start with abc
  • (regardless of chunk mode) while retrieving indexes, keep-balance logs "index progress: received {count} blocks from {keepstore-mount-url}" every 5 minutes
  • (regardless of chunk mode) during a balancing operation, keep-balance checks its process memory size once per second, and reports it in a log message if, since the last report, it has grown 40% or 10 minutes have passed
Actions #4

Updated by Lucas Di Pentima 1 day ago

Some questions:

  • I think it would be useful to have validations on --chunk-prefix param. If they were skipped because it's an experimental feature, please disregard this comment.
  • I don't understand the utility of sleeping one second on services/keep-balance/balance.go line 159, is this something that leaked from testing?
  • Is it possible to get mem usage stats from https://pkg.go.dev/runtime#MemStats ? I'm thinking that if we can avoid reading the /proc fs, it's a step forward to be possible to use keep-balance on different OSes.
Actions #5

Updated by Tom Clegg 1 day ago

I think sanity-checking the prefix is a good idea -- the most likely error is passing uppercase hex digits, which would confusingly never match anything. Added.

Oops, the sleep was indeed just for testing. Removed.

I did consider using the Go runtime features to track memory use, but went with procfs instead so that (if there's any difference at all) the stats will match "top" etc.

The portability point is fair, though. Another reason to use the Go runtime is that the operator already has the option to watch "top" if they just want the OS-reported numbers. What's better -- try procfs first and fall back on the Go runtime? Or just use the Go runtime?

Another point: the rss figure might also be interesting, but in the context of tracking/predicting the resources needed to do the job, I thought vsize would be more relevant -- if pages are being swapped out, rss will appear low, which seems misleading. Does this sound right?

19923-keep-balance-chunk @ 1ddc21430ede024aa6841e6c49deafe87788c287 -- developer-run-tests: #3462

Actions #6

Updated by Lucas Di Pentima 1 day ago

Tom Clegg wrote in #note-5:

The portability point is fair, though. Another reason to use the Go runtime is that the operator already has the option to watch "top" if they just want the OS-reported numbers. What's better -- try procfs first and fall back on the Go runtime? Or just use the Go runtime?

Using procfs as a first option is a nice idea, so that there's no difference between what the operator sees in "top" and what's reported in the logs.

Another point: the rss figure might also be interesting, but in the context of tracking/predicting the resources needed to do the job, I thought vsize would be more relevant -- if pages are being swapped out, rss will appear low, which seems misleading. Does this sound right?

If it isn't too much work having both reported, I think it would be useful in cases where keep-balance is being run on a shared node and getting OOMed intermittently because of some other memory hungry process.

The rest LGTM.

Actions #7

Updated by Tom Clegg 1 day ago

This update
  • reports HeapInuse, HeapAlloc, and StackInuse from the Go runtime
  • reports virt and res from procfs if available
  • bases the "increased by 40%" trigger on heapInuse+stackInuse

19923-keep-balance-chunk @ 7d3a6de4ee5afb635e561c75d4f99d1067f030f2 -- developer-run-tests: #3464

Actions #8

Updated by Lucas Di Pentima 1 day ago

LGTM, thanks!

Actions #9

Updated by Tom Clegg about 22 hours ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF