Project

General

Profile

Actions

Idea #20693

closed

Design for server side coordination of multiple writers to a collection

Added by Peter Amstutz over 1 year ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Start date:
05/24/2024
Due date:
Story points:
-

Description

Background:

Multiple Arvados services (multiple instances of keep-web, arvados-client mount, arv-mount, etc) are trying to write files to the same collection at the same time.

Assume they are adding/removing/changing multiple files but not making changes that directly conflict/contradict one another.

Requirements:

  • If a file is created or modified it won't disappear as a result of an update from another service that didn't know about that file
  • If there is a single writer, performance impact should be minimal
  • If there are multiple writers, it is acceptable that one of them may have to wait to avoid conflicts
  • Can use pessimistic locking to ensure only one client can have a write lock at a time, attempting to open a file for writing that is locked by another should return an error on open
  • Support WebDAV lock protocol

Proposal: Concurrent writes to a single collection


Subtasks 1 (0 open1 closed)

Task #21625: Review at engineering meetingResolved05/24/2024Actions

Related issues

Related to Arvados Epics - Idea #18342: Keep performance optimizationNew08/01/202305/30/2024Actions
Actions #1

Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz over 1 year ago

  • Status changed from In Progress to New
  • Tracker changed from Bug to Feature
Actions #3

Updated by Peter Amstutz over 1 year ago

  • Category set to Keep
Actions #4

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Related to Idea #18342: Keep performance optimization added
Actions #6

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Future to Development 2024-01-17 sprint
Actions #7

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-01-03 sprint
Actions #8

Updated by Peter Amstutz 12 months ago

  • Target version changed from Development 2024-01-03 sprint to Development 2024-01-17 sprint
Actions #9

Updated by Peter Amstutz 10 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-02-14 sprint
Actions #10

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-02-14 sprint to Development 2024-02-28 sprint
Actions #11

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-03-13 sprint
Actions #12

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-03-13 sprint to Development 2024-03-27 sprint
Actions #13

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Actions #14

Updated by Peter Amstutz 7 months ago

  • Subject changed from Server side coordination of multiple writers to a collection to Design for server side coordination of multiple writers to a collection
  • Tracker changed from Feature to Idea
Actions #15

Updated by Peter Amstutz 7 months ago

  • Assigned To set to Tom Clegg
Actions #16

Updated by Tom Clegg 6 months ago

  • Target version changed from Development 2024-04-10 sprint to Development 2024-04-24 sprint
Actions #17

Updated by Tom Clegg 6 months ago

  • Status changed from New to In Progress
Actions #18

Updated by Tom Clegg 6 months ago

  • Description updated (diff)
Actions #19

Updated by Peter Amstutz 6 months ago

  • Target version changed from Development 2024-04-24 sprint to Development 2024-05-08 sprint
Actions #20

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-05-08 sprint to Development 2024-05-22 sprint
Actions #21

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-05-22 sprint to Development 2024-06-05 sprint
Actions #22

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-06-05 sprint to Development 2024-06-19 sprint
Actions #23

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-06-19 sprint to Development 2024-07-03 sprint
Actions #24

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-07-03 sprint to Development 2024-07-24 sprint
Actions #25

Updated by Peter Amstutz 3 months ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF