Bug #16425

[keepstore] should not scan directories it doesn't write in

Added by Ward Vandewege over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
06/12/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Keepstore walks all directories in its mount points, even those it never writes in. This can lead to a situation where keepstore does tons of unnecessary IO (e.g. when .snapshot directories are present, where GPFS makes daily snapshots).

Adding an option to exclude certain directories from what Keep considers would be a good solution.


Subtasks

Task #16487: Review 16425-skip-unused-dirsResolvedWard Vandewege

Associated revisions

Revision 2cae80e7
Added by Tom Clegg over 1 year ago

Merge branch '16425-skip-unused-dirs'

fixes #16425

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Ward Vandewege over 1 year ago

  • Description updated (diff)

#2 Updated by Ward Vandewege over 1 year ago

  • Description updated (diff)

#3 Updated by Tom Clegg over 1 year ago

When indexing, it looks like we're already skipping dirs that don't conform to keepstore's storage layout: source:services/keepstore/unix_volume.go#L355

var blockDirRe = regexp.MustCompile(`^[0-9a-f]+$`)
var blockFileRe = regexp.MustCompile(`^[0-9a-f]{32}$`)
...
                if !blockDirRe.MatchString(names[0]) {
                        continue
                }
                ...
                blockdir, err := v.os.Open(blockdirpath)

However, the "empty trash" goroutine does walk the entire tree.

        err := filepath.Walk(v.Root, func(path string, info os.FileInfo, err error) error {
                if err != nil {
                        v.logger.WithError(err).Errorf("EmptyTrash: filepath.Walk(%q) failed", path)
                        return nil
                }
                todo <- dirent{path, info}
                return nil
        })

That walk func should check dirnames and return filepath.SkipDir when appropriate.

#4 Updated by Peter Amstutz over 1 year ago

  • Target version changed from To Be Groomed to 2020-06-17 Sprint
  • Assigned To set to Tom Clegg

#6 Updated by Tom Clegg over 1 year ago

  • Status changed from New to In Progress

#8 Updated by Anonymous over 1 year ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#9 Updated by Peter Amstutz about 1 year ago

  • Release set to 25

Also available in: Atom PDF