Story #1968

Monitor disk usage (per user and per site; split by transient/persistent; optionally weighted by #users who want persistent).

Added by Tom Clegg almost 7 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
Misha Zatsman
Category:
-
Start date:
04/09/2014
Due date:
% Done:

100%

Estimated time:
(Total: 4.00 h)
Story points:
2.0
Release relationship:
Auto

Description

Focus on
  • Showing usage in Workbench
  • Preliminary garbage collection results
Targets
  • Set up a data manager service that generates reports for qr1hi periodically.
  • Deliver reports as JSON in Log objects. (At least while our metadata/event/log systems are being sorted out.)
  • Show on Workbench
    • Workbench "user list" / "group list" pages will show latest usage numbers for each user/group
    • Workbench "show user" / "show group" pages will show last few usage reports for that user/group
  • Garbage collection & replication-checking results -- some attainable subset of...
    • Report statistics (blobs with less/more than required replication)
    • Report hashes of blobs with less than required replication
    • Next candidates for deletion (oldest not mentioned in any wanted collection)
    • Report cache window vs. % free space: "we can get 30% free space by deleting everything >54 days old" (this graph might conceivably show interesting usage patterns)
    • Candidates appear in which unwanted collections (if any)? Which of those collections are currently intact (i.e., will get their first "holes" in next GC run)
    • (?) Report of affected collections for each user (but we'd need "replication=0" instead of "don't want" in order to know this, right?)

Subtasks

Task #2572: Report weighted disk usage based on users' desired replication.ResolvedMisha Zatsman

Task #2571: Read (and store) replication preferences from "want" link properties.ResolvedMisha Zatsman

Task #2590: Report deletion candidates sorted by age.ResolvedMisha Zatsman

Task #2574: Report blocks that are replicated less than the max desired by all persisters.ResolvedMisha Zatsman

Task #2579: Starting logging to the API server,ResolvedMisha Zatsman

Task #2573: Modify reader disk usage numbers to not consider disk replication.ResolvedMisha Zatsman

Task #1757: User can see in Workbench total amount storage space being usedResolved

History

#1 Updated by Peter Amstutz almost 7 years ago

  • Story points set to 2.0

#2 Updated by Ward Vandewege over 6 years ago

  • Target version set to 2014-04-16 Dev tools and data/resource management

#3 Updated by Tom Clegg over 6 years ago

  • Assigned To set to Misha Zatsman

#4 Updated by Tom Clegg over 6 years ago

  • Description updated (diff)

#5 Updated by Tom Clegg over 6 years ago

  • Description updated (diff)

#6 Updated by Tom Clegg over 6 years ago

  • Description updated (diff)

#7 Updated by Tom Clegg over 6 years ago

  • Status changed from New to In Progress

#8 Updated by Tom Clegg over 6 years ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF