Feature #21351
Updated by Peter Amstutz 4 months ago
User has 75000000 objects in one S3 bucket. It takes about 5 hours to list all those objects in order to run keep-balance. We should think about how to make this go faster.
AWS provides an alternative to the list API to getting the list of objects in a bucket, called S3 Inventory:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html
With S3 Inventory, we could download a single flat file with all the objects listed. The file is updated daily.
-- Although, on second thought, if that means we can only run keep-balance once a day, it ends up being a wash.