Project

General

Profile

Actions

Feature #13383

closed

[keepstore] configurable number of trash workers

Added by Tom Clegg over 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
0.5
Release:
Release relationship:
Auto

Description

Currently keepstore works the "trash list" with a single thread. On some platforms (AWS S3) each delete operation can take multiple seconds, which makes garbage collection very slow.

Everything is already set up for multiple goroutines to work the trash and pull queues. We just need a config knob, and a bit of code to start N goroutines instead of just one at startup.


Subtasks 1 (0 open1 closed)

Task #13441: Review 13383-trash-workersResolvedTom Clegg05/04/2018Actions
Actions #2

Updated by Tom Clegg over 6 years ago

Tentative code at 13335-trash-workers @ 10803caaaf1b59caf5e4c49f78eca71685f1c4ae

diff --git a/services/keepstore/config.go b/services/keepstore/config.go
index 17d6acdb6..19dc7f69b 100644
--- a/services/keepstore/config.go
+++ b/services/keepstore/config.go
@@ -40,6 +40,9 @@ type Config struct {
        EnableDelete        bool
        TrashLifetime       arvados.Duration
        TrashCheckInterval  arvados.Duration
+       PullWorkers         int
+       TrashWorkers        int
+       EmptyTrashWorkers   int

        Volumes VolumeList

Actions #3

Updated by Tom Morris over 6 years ago

  • Target version changed from To Be Groomed to 2018-05-09 Sprint
  • Story points set to 0.5
Actions #4

Updated by Tom Clegg over 6 years ago

  • Status changed from New to In Progress
  • Assigned To set to Tom Clegg
Actions #5

Updated by Tom Clegg over 6 years ago

Actions #6

Updated by Lucas Di Pentima over 6 years ago

  • Is concurrency really needed on the filesystem backend if it’s per-volume?
  • The rest LGTM
Actions #7

Updated by Tom Clegg over 6 years ago

Lucas Di Pentima wrote:

  • Is concurrency really needed on the filesystem backend if it’s per-volume?

It does seem unlikely to be needed with local disks, but the filesystem backend can also be used to connect networked/distributed filesystems so I thought it might be worth having.

Actions #8

Updated by Tom Clegg over 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100
Actions #9

Updated by Tom Morris about 6 years ago

  • Release set to 13
Actions

Also available in: Atom PDF