Feature #16745

[keep-web] Improve performance of S3 APIs using server-side cache

Added by Tom Clegg about 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Keep
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Currently, when handling a series of S3 API calls, keep-web builds a new filesystem object for each call. In some common workflows (reading several small files from a large collection / deep in a project tree), this will be very inefficient.

Initial implementation may be a map[apitoken]sitefs, using a single sitefs to serve many read-only requests until reaching a configured TTL or performing a write operation with the same token. This should be relatively simple, and although write operations would still be inefficient, sequences of read operations would be much faster.


Related issues

Related to Arvados Epics - Story #16360: Keep-web supports S3 compatible interfaceIn Progress07/01/202011/18/2020

History

#1 Updated by Tom Clegg about 2 months ago

  • Related to Story #16360: Keep-web supports S3 compatible interface added

Also available in: Atom PDF