Arvados

User has 75 million objects in one S3 bucket. It takes about 5 hours to list all those objects in order to run keep-balance. We should think about how to make this go faster.

AWS provides an alternative to the list API to getting the list of objects in a bucket, called S3 Inventory:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

With S3 Inventory, we could download a single flat file with all the objects listed. The file is updated daily.

-- Although, on second thought, if that means we can only run keep-balance once a day, it ends up being a wash.