Project

General

Profile

Actions

Feature #20318

open

Go SDK supports local filesystem-backed data cache

Added by Peter Amstutz 8 months ago. Updated 1 day ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
Keep
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Add option to store cached blocks in local filesystem rather than RAM in order to take advantage of better performance of (typically SSD-backed) local disk relative to backend, and let the kernel decide what to cache in RAM.

Where reasonable, implement the same behavior as the Python SDK in laying out and managing the cache directory containing the files for each cached block.

Caller should be able to read a partial block before the entire block is received from upstream, allowing for streaming block content. Can base this on AsyncBuf but uses a file or mmap as the backing store instead of a plain slice.


Subtasks 1 (1 open0 closed)

Task #21177: ReviewNewLucas Di Pentima

Actions

Related issues

Related to Arvados Epics - Story #18342: Keep performance optimizationNew08/01/202310/31/2023

Actions
Related to Arvados - Feature #21196: implement keepclient API on top of block cache with pluggable backendNew

Actions
Actions #1

Updated by Peter Amstutz 5 months ago

  • Related to Story #17001: Arvados uses WB2 by default added
Actions #2

Updated by Peter Amstutz 5 months ago

  • Related to deleted (Story #17001: Arvados uses WB2 by default)
Actions #3

Updated by Peter Amstutz 5 months ago

  • Related to Story #18342: Keep performance optimization added
Actions #4

Updated by Peter Amstutz 5 months ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz 2 months ago

  • Target version changed from To be groomed to Development 2023-11-08 sprint
Actions #6

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #7

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2023-11-29 sprint to Development 2024-01-03 sprint
Actions #8

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-01-03 sprint to Development 2023-11-29 sprint
Actions #9

Updated by Tom Clegg 2 months ago

  • Description updated (diff)
  • Subject changed from Go SDK keep cache supports memory mapped block cache to Go SDK supports local filesystem-backed data cache
Actions #10

Updated by Peter Amstutz about 2 months ago

  • Target version changed from Development 2023-11-29 sprint to Development 2023-10-25 sprint
Actions #11

Updated by Peter Amstutz about 2 months ago

  • Assigned To set to Tom Clegg
Actions #12

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Actions #13

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #14

Updated by Peter Amstutz 15 days ago

  • Description updated (diff)
Actions #15

Updated by Peter Amstutz 15 days ago

  • Related to Feature #21196: implement keepclient API on top of block cache with pluggable backend added
Actions #16

Updated by Peter Amstutz 2 days ago

  • Status changed from New to In Progress
Actions #17

Updated by Peter Amstutz 1 day ago

  • Target version changed from Development 2023-11-29 sprint to Development 2024-01-03 sprint
Actions

Also available in: Atom PDF