Project

General

Profile

Actions

Feature #16745

closed

[keep-web] Improve performance of S3 APIs using server-side cache

Added by Tom Clegg over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Story points:
-
Release relationship:
Auto

Description

Currently, when handling a series of S3 API calls, keep-web builds a new filesystem object for each call. In some common workflows (reading several small files from a large collection / deep in a project tree), this will be very inefficient.

Initial implementation may be a map[apitoken]sitefs, using a single sitefs to serve many read-only requests until reaching a configured TTL or performing a write operation with the same token. This should be relatively simple, and although write operations would still be inefficient, sequences of read operations would be much faster.


Subtasks 1 (0 open1 closed)

Task #17379: Review 16745-keep-web-cacheResolvedTom Clegg03/02/2021Actions

Related issues 1 (0 open1 closed)

Related to Arvados Epics - Idea #16516: Run Keepstore on local compute nodesResolved10/01/202111/30/2021Actions
Actions

Also available in: Atom PDF