Project

General

Profile

Actions

Idea #20996

closed

Efficient packing of small files into blocks in keep-web

Added by Peter Amstutz about 1 year ago. Updated 7 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
11/13/2024
Due date:
Story points:
-

Description

Propose a design for block packing in keep-web based on either a configurable delay or lock/unlock operation, so that writing multiple small files to the same collection within that delay buffers and/or automatically repacks data for multiple small files into the same data block. This may have the side effect that the collection is updated multiple times as files are added and blocks are repacked.

Discuss design at engineering meeting.

See proposal at Efficient block packing for small WebDAV uploads


Subtasks 1 (0 open1 closed)

Task #22254: review at engineering meetingResolved11/13/2024Actions

Related issues

Related to Arvados Epics - Idea #18342: Keep performance optimizationNew08/01/202312/31/2024Actions
Related to Arvados - Feature #22321: Update keepweb.handler to repack after writingNewTom CleggActions
Related to Arvados - Feature #22320: Add Repack(opts RepackOptions) method to collectionfs, dirnode, and filehandleNewTom CleggActions
Related to Arvados - Feature #22319: Add replace_segments feature to controller's CollectionUpdate APIIn ProgressTom CleggActions
Actions

Also available in: Atom PDF