Feature #18961


Go FileSystem / FUSE mount supports block prefetch

Added by Peter Amstutz about 2 years ago. Updated about 1 month ago.

Assigned To:
Story points:


Use the following strategy for prefetch:

When a read happens on a file, look at the next N blocks the make up the manifest stream and issue prefetch requests for those blocks. These blocks get loaded into the cache so they are ready to go when they are needed.

By looking ahead in the stream rather than just the file, this also works for manifests containing small files which are stored as 1 block per file.

There should be a config knob to control how much data or blocks are prefetched so that sites can experiment with optimal throughput.

This implies the cache behavior needs to support pre-fetch, which means pre-fetched blocks should not push out actively used blocks but should be able to push out less recently used blocks. Plain LRU behavior where a block is promoted each time it is accessed may be sufficient but metrics will be helpful.


download-speed-1.8-GB-file.png (11.6 KB) download-speed-1.8-GB-file.png Tom Clegg, 02/20/2024 02:48 PM
download-speed-1.8-GB-file-2.png (29.7 KB) download-speed-1.8-GB-file-2.png Tom Clegg, 02/26/2024 02:54 PM
download-speed-1800-1MB-files.png (22.8 KB) download-speed-1800-1MB-files.png Tom Clegg, 02/26/2024 08:03 PM
download-speed-1.8-GB-file-4.png (24.8 KB) download-speed-1.8-GB-file-4.png Tom Clegg, 02/26/2024 09:26 PM
smol.png (20.1 KB) smol.png Tom Clegg, 02/27/2024 02:55 AM
smol2.png (21.2 KB) smol2.png Tom Clegg, 02/27/2024 03:40 PM
smol3.png (22.1 KB) smol3.png Tom Clegg, 02/27/2024 04:21 PM
smol4.png (30.5 KB) smol4.png Tom Clegg, 02/27/2024 05:45 PM

Related issues

Related to Arvados Epics - Idea #17849: FUSE driver v2NewActions
Related to Arvados Epics - Idea #18342: Keep performance optimizationNew08/01/202305/30/2024Actions
Related to Arvados - Feature #20995: Prefetch small files when scanning a collection directoryDuplicateTom CleggActions
Related to Arvados - Feature #21606: configurable keep-web output buffer to reduce delay between blocksResolvedTom CleggActions

Also available in: Atom PDF