Project

General

Profile

Actions

Bug #9363

open

[keep-balance] Avoid deleting recently-referenced blocks

Added by Tom Clegg almost 8 years ago. Updated 26 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Keep
Target version:
Story points:
1.0
Release:
Release relationship:
Auto

Description

The following sequence can break the invariant Keep relies on for safe garbage collection1:
  1. Write data blocks
  2. Save collection
  3. Wait for original blob signature TTLs (from step 1) to expire
  4. Retrieve the collection (setting aside a copy in memory)
  5. Update the collection to remove references to some (or all) blocks
  6. Wait for keep-balance to do a garbage-collection sweep
  7. Update the collection (or save a new one) using the blocks/signatures set aside in step 4

If the only reference to a block is the one in this collection, the block may be deleted during step 6, resulting in a collection being saved successfully in step 7 even though some of its data is missing.

Proposed solution:

When updating a collection, detect block references that are being dropped2; if there are any, save a temporary collection (trashed, readable only by admin) referencing all of the dropped blocks. These will be cleaned up when they expire, and in the meantime they will be seen by keep-balance, thus protecting them until even their new signatures (issued in step 4) expire.

1 If any client has a signed block locator with a future timestamp, that block either [i] is newer than BlobSignatureTTL according to the timestamp stored on the backend, and therefore won't be deleted; or [ii] appears in a collection record where keep-balance can see it, and therefore won't be deleted.

2 This might be done during the (existing) strip_signatures_and_update_replication_confirmed hook where we detect whether references to new blocks have been added, so we know whether to reset the replication_confirmed columns.


Related issues

Related to Arvados - Feature #4650: [API] API method and CLI shortcut for refreshing the signatures on some block locators (without creating a collection)NewActions
Actions

Also available in: Atom PDF