Project

General

Profile

Actions

Feature #15125

open

[keep-balance] [keepstore] Procedure to halt/reverse/investigate a suspected data loss incident

Added by Tom Clegg over 5 years ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

A site admin, upon suspecting keep-balance is erroneously trashing some data, should be able to
  • act quickly to minimize the impact, and
  • characterize the damage, if any
Steps to minimize the impact:
  • immediately prevent keepstore from trashing or deleting any blocks while investigation/recovery proceeds
  • untrash any blocks that might have been trashed erroneously (this may enable affected workflows to resume)
Steps to characterize the damage:
  • get a list of missing block IDs
  • get a list of collections that reference missing blocks (including uuid, pdh, name, project uuid, project name)
Troubleshooting:
  • report version in metrics (e.g., version{program="keep-balance", version="1.3.1"} = 1)
  • report #+size of trashed blocks in metrics
  • keepstore "untrash all" management API
  • keep-balance reporting option to get debug info for a list of specific collection IDs and block IDs (without getting the entire debug dump, which is huge)
  • keep-block-check --collection=uuid_or_pdh

Related issues

Related to Arvados Epics - Idea #16514: Actionable insight into keep usageNewActions
Actions #2

Updated by Tom Clegg over 5 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Morris over 5 years ago

  • Target version set to To Be Groomed
Actions #4

Updated by Ward Vandewege over 4 years ago

  • Related to Idea #16514: Actionable insight into keep usage added
Actions #5

Updated by Peter Amstutz over 3 years ago

  • Target version deleted (To Be Groomed)
Actions #6

Updated by Peter Amstutz over 1 year ago

  • Release set to 60
Actions #7

Updated by Peter Amstutz 8 months ago

  • Target version set to Future
Actions

Also available in: Atom PDF