Feature #13383
closed
[keepstore] configurable number of trash workers
Added by Tom Clegg over 6 years ago.
Updated over 6 years ago.
Release relationship:
Auto
Description
Currently keepstore works the "trash list" with a single thread. On some platforms (AWS S3) each delete operation can take multiple seconds, which makes garbage collection very slow.
Everything is already set up for multiple goroutines to work the trash and pull queues. We just need a config knob, and a bit of code to start N goroutines instead of just one at startup.
Tentative code at 13335-trash-workers @ 10803caaaf1b59caf5e4c49f78eca71685f1c4ae
diff --git a/services/keepstore/config.go b/services/keepstore/config.go
index 17d6acdb6..19dc7f69b 100644
--- a/services/keepstore/config.go
+++ b/services/keepstore/config.go
@@ -40,6 +40,9 @@ type Config struct {
EnableDelete bool
TrashLifetime arvados.Duration
TrashCheckInterval arvados.Duration
+ PullWorkers int
+ TrashWorkers int
+ EmptyTrashWorkers int
Volumes VolumeList
- Target version changed from To Be Groomed to 2018-05-09 Sprint
- Story points set to 0.5
- Status changed from New to In Progress
- Assigned To set to Tom Clegg
- Is concurrency really needed on the filesystem backend if it’s per-volume?
- The rest LGTM
Lucas Di Pentima wrote:
- Is concurrency really needed on the filesystem backend if it’s per-volume?
It does seem unlikely to be needed with local disks, but the filesystem backend can also be used to connect networked/distributed filesystems so I thought it might be worth having.
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Also available in: Atom
PDF
Updated by 
Updated by Tom Morris 
Updated by