Project

General

Profile

Actions

Feature #22471

open

[keep-balance] multi-pass mode

Added by Tom Clegg about 1 month ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Story points:
-

Description

  • When posting a new trash/pull list, set X-Keep-List-Prefix header, so keepstore knows which entries to clear
  • Run a pass for each prefix, then merge resulting statistics to produce full cluster summary
  • when starting a multi-pass sweep, clear replication_confirmed_partial
  • after each pass (single prefix), set or reduce replication_confirmed_partial
  • after all passes (prefixes) are done, copy replication_confirmed_partial to replication_confirmed

see Multi-pass mode to reduce keep-balance memory footprint


Related issues 3 (2 open1 closed)

Related to Arvados - Idea #22218: design for horizontal scaling of Keep-balance to handle increasingly large clustersResolvedTom Clegg01/16/2025Actions
Blocked by Arvados - Feature #22469: [keepstore] support multi-pass modeNewTom CleggActions
Blocked by Arvados - Feature #22470: [railsapi] add replication_confirmed_partial field to collectionsNewTom CleggActions
Actions #1

Updated by Tom Clegg about 1 month ago

  • Blocked by Feature #22469: [keepstore] support multi-pass mode added
Actions #2

Updated by Tom Clegg about 1 month ago

  • Blocked by Feature #22470: [railsapi] add replication_confirmed_partial field to collections added
Actions #3

Updated by Tom Clegg 21 days ago

  • Related to Idea #22218: design for horizontal scaling of Keep-balance to handle increasingly large clusters added
Actions

Also available in: Atom PDF