Feature #21351
Updated by Peter Amstutz 11 months ago
User has 75 million 75000000 objects in one S3 bucket. It takes about 5 hours to list all those objects in order to run keep-balance. We should think about how to make this go faster. AWS provides an alternative to the list API to getting the list of objects in a bucket, called S3 Inventory: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html With S3 Inventory, we could download a single flat file with all the objects listed. The file is updated daily. -- Although, on second thought, if that means we can only run keep-balance once a day, it ends up being a wash.