Project

General

Profile

Actions
Copy link

Idea #6311

closed

[Maybe] [SDKs] Support caching Keep blocks in memcached

Added by Brett Smith almost 9 years ago. Updated about 5 years ago.

Status:
Rejected
Priority:
Normal
Assigned To:
-
Category:
SDKs
Target version:
-
Start date:
Due date:
Story points:
-

Description

We could potentially improve job performance by running memcached on each compute node to store Keep blocks. When a node is running many tasks from a job that access the same data, this cache could make it possible for the block to be downloaded to the node once, then shared across tasks.

If we decide to go ahead with this caching strategy, add the necessary support to the Python SDK Keepclient to use a memcached store when available.

Related issues

Related to Arvados - Idea #3640: [SDKs] Add runtime option to SDKs (esp Python and arv-mount) to use a filesystem directory block cache as an alternative to RAM cache.ClosedActions
Actions
Copy link

Also available in: Atom PDF