Arvados

Want to have estimates of storage usage.

Proposal:

Keep-balance keeps track of storage usage per-project, and updates the project records at the conclusion of each run, similarly to how it currently maintains the replication status fields for collections.

For each project, it should record:

• Perceived usage - approximately what you would get from sum(file_size_total) of collections in the project
• Project deduplicated usage - each unique block counted exactly once
• Whole cluster deduplicated usage - each unique block size multiplied by the ratio of (appearances of that block in project / appearances in whole cluster)