Feature #13383
closed
[keepstore] configurable number of trash workers
Description
Currently keepstore works the "trash list" with a single thread. On some platforms (AWS S3) each delete operation can take multiple seconds, which makes garbage collection very slow.
Everything is already set up for multiple goroutines to work the trash and pull queues. We just need a config knob, and a bit of code to start N goroutines instead of just one at startup.
Updated by Tom Clegg almost 6 years ago
Tentative code at 13335-trash-workers @ 10803caaaf1b59caf5e4c49f78eca71685f1c4ae
diff --git a/services/keepstore/config.go b/services/keepstore/config.go
index 17d6acdb6..19dc7f69b 100644
--- a/services/keepstore/config.go
+++ b/services/keepstore/config.go
@@ -40,6 +40,9 @@ type Config struct {
EnableDelete bool
TrashLifetime arvados.Duration
TrashCheckInterval arvados.Duration
+ PullWorkers int
+ TrashWorkers int
+ EmptyTrashWorkers int
Volumes VolumeList
Updated by Tom Morris almost 6 years ago
- Target version changed from To Be Groomed to 2018-05-09 Sprint
- Story points set to 0.5
Updated by Tom Clegg almost 6 years ago
- Status changed from New to In Progress
- Assigned To set to Tom Clegg
Updated by Tom Clegg almost 6 years ago
13383-trash-workers @ dcb4760843cc0ed4647e8eaa43abb5d2f049cd0c
Updated by Lucas Di Pentima almost 6 years ago
- Is concurrency really needed on the filesystem backend if it’s per-volume?
- The rest LGTM
Updated by Tom Clegg almost 6 years ago
Lucas Di Pentima wrote:
- Is concurrency really needed on the filesystem backend if it’s per-volume?
It does seem unlikely to be needed with local disks, but the filesystem backend can also be used to connect networked/distributed filesystems so I thought it might be worth having.
Updated by Tom Clegg almost 6 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|0d50e82dd2255104e60c0882045b54774e1be380.