Feature #16745

[keep-web] Improve performance of S3 APIs using server-side cache

Added by Tom Clegg 5 months ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


Currently, when handling a series of S3 API calls, keep-web builds a new filesystem object for each call. In some common workflows (reading several small files from a large collection / deep in a project tree), this will be very inefficient.

Initial implementation may be a map[apitoken]sitefs, using a single sitefs to serve many read-only requests until reaching a configured TTL or performing a write operation with the same token. This should be relatively simple, and although write operations would still be inefficient, sequences of read operations would be much faster.

Related issues

Related to Arvados Epics - Story #16516: Keep performanceNew08/01/202110/31/2021


#1 Updated by Tom Clegg 5 months ago

  • Related to Story #16360: Keep-web supports S3 compatible interface added

#2 Updated by Peter Amstutz about 1 month ago

#3 Updated by Peter Amstutz about 1 month ago

  • Related to deleted (Story #16360: Keep-web supports S3 compatible interface)

Also available in: Atom PDF