Feature #13383

[keepstore] configurable number of trash workers

Added by Tom Clegg almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
05/04/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5
Release:
Release relationship:
Auto

Description

Currently keepstore works the "trash list" with a single thread. On some platforms (AWS S3) each delete operation can take multiple seconds, which makes garbage collection very slow.

Everything is already set up for multiple goroutines to work the trash and pull queues. We just need a config knob, and a bit of code to start N goroutines instead of just one at startup.


Subtasks

Task #13441: Review 13383-trash-workersResolvedTom Clegg

Associated revisions

Revision 0d50e82d
Added by Tom Clegg almost 3 years ago

Merge branch '13383-trash-workers'

closes #13383
refs #13335

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#2 Updated by Tom Clegg almost 3 years ago

Tentative code at 13335-trash-workers @ 10803caaaf1b59caf5e4c49f78eca71685f1c4ae

diff --git a/services/keepstore/config.go b/services/keepstore/config.go
index 17d6acdb6..19dc7f69b 100644
--- a/services/keepstore/config.go
+++ b/services/keepstore/config.go
@@ -40,6 +40,9 @@ type Config struct {
        EnableDelete        bool
        TrashLifetime       arvados.Duration
        TrashCheckInterval  arvados.Duration
+       PullWorkers         int
+       TrashWorkers        int
+       EmptyTrashWorkers   int

        Volumes VolumeList

#3 Updated by Tom Morris almost 3 years ago

  • Target version changed from To Be Groomed to 2018-05-09 Sprint
  • Story points set to 0.5

#4 Updated by Tom Clegg almost 3 years ago

  • Status changed from New to In Progress
  • Assigned To set to Tom Clegg

#6 Updated by Lucas Di Pentima almost 3 years ago

  • Is concurrency really needed on the filesystem backend if it’s per-volume?
  • The rest LGTM

#7 Updated by Tom Clegg almost 3 years ago

Lucas Di Pentima wrote:

  • Is concurrency really needed on the filesystem backend if it’s per-volume?

It does seem unlikely to be needed with local disks, but the filesystem backend can also be used to connect networked/distributed filesystems so I thought it might be worth having.

#8 Updated by Tom Clegg almost 3 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

#9 Updated by Tom Morris over 2 years ago

  • Release set to 13

Also available in: Atom PDF