Project

General

Profile

Actions

Idea #20996

closed

Efficient packing of small files into blocks in keep-web

Added by Peter Amstutz about 1 year ago. Updated 7 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
11/13/2024
Due date:
Story points:
-

Description

Propose a design for block packing in keep-web based on either a configurable delay or lock/unlock operation, so that writing multiple small files to the same collection within that delay buffers and/or automatically repacks data for multiple small files into the same data block. This may have the side effect that the collection is updated multiple times as files are added and blocks are repacked.

Discuss design at engineering meeting.

See proposal at Efficient block packing for small WebDAV uploads


Subtasks 1 (0 open1 closed)

Task #22254: review at engineering meetingResolved11/13/2024Actions

Related issues

Related to Arvados Epics - Idea #18342: Keep performance optimizationNew08/01/202312/31/2024Actions
Related to Arvados - Feature #22321: Update keepweb.handler to repack after writingNewTom CleggActions
Related to Arvados - Feature #22320: Add Repack(opts RepackOptions) method to collectionfs, dirnode, and filehandleNewTom CleggActions
Related to Arvados - Feature #22319: Add replace_segments feature to controller's CollectionUpdate APIIn ProgressTom CleggActions
Actions #1

Updated by Peter Amstutz about 1 year ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz about 1 year ago

  • Status changed from In Progress to New
Actions #3

Updated by Peter Amstutz about 1 year ago

  • Description updated (diff)
Actions #4

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-02-28 sprint
Actions #5

Updated by Peter Amstutz 9 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-03-13 sprint
Actions #6

Updated by Peter Amstutz 9 months ago

  • Target version changed from Development 2024-03-13 sprint to Development 2024-03-27 sprint
Actions #7

Updated by Peter Amstutz 9 months ago

  • Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Actions #8

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-04-10 sprint to Development 2024-04-24 sprint
Actions #9

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-04-24 sprint to Development 2024-05-08 sprint
Actions #10

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-05-08 sprint to Development 2024-06-05 sprint
Actions #11

Updated by Peter Amstutz 6 months ago

  • Target version changed from Development 2024-06-05 sprint to 439
Actions #12

Updated by Peter Amstutz 6 months ago

  • Target version changed from 439 to Development 2024-07-03 sprint
Actions #13

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-07-03 sprint to Development 2024-07-24 sprint
Actions #14

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-07-24 sprint to Development 2024-08-28 sprint
Actions #15

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-08-28 sprint to Development 2024-09-11 sprint
Actions #16

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-09-11 sprint to Development 2024-08-28 sprint
Actions #17

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-08-28 sprint to Development 2024-09-11 sprint
Actions #18

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-09-11 sprint to Development 2024-09-25 sprint
Actions #19

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-09-25 sprint to Development 2024-10-09 sprint
Actions #20

Updated by Peter Amstutz 2 months ago

  • Tracker changed from Feature to Idea
Actions #21

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-10-09 sprint to Development 2024-10-23 sprint
Actions #22

Updated by Peter Amstutz about 2 months ago

  • Target version changed from Development 2024-10-23 sprint to Development 2024-11-06 sprint
Actions #23

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2024-11-06 sprint to Development 2024-10-23 sprint
Actions #24

Updated by Tom Clegg about 1 month ago

  • Assigned To set to Tom Clegg
Actions #25

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2024-10-23 sprint to Development 2024-11-06 sprint
Actions #26

Updated by Peter Amstutz 29 days ago

  • Description updated (diff)
Actions #27

Updated by Peter Amstutz 15 days ago

  • Target version changed from Development 2024-11-06 sprint to Development 2024-11-20
Actions #28

Updated by Tom Clegg 13 days ago

  • Related to Idea #18342: Keep performance optimization added
Actions #29

Updated by Tom Clegg 13 days ago

  • Status changed from New to In Progress
  • Description updated (diff)
Actions #30

Updated by Tom Clegg 8 days ago

  • Related to Feature #22321: Update keepweb.handler to repack after writing added
Actions #31

Updated by Tom Clegg 8 days ago

  • Related to Feature #22320: Add Repack(opts RepackOptions) method to collectionfs, dirnode, and filehandle added
Actions #32

Updated by Tom Clegg 8 days ago

  • Related to Feature #22319: Add replace_segments feature to controller's CollectionUpdate API added
Actions #33

Updated by Tom Clegg 8 days ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF