Project

General

Profile

Actions

Bug #16425

closed

[keepstore] should not scan directories it doesn't write in

Added by Ward Vandewege almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
06/12/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Keepstore walks all directories in its mount points, even those it never writes in. This can lead to a situation where keepstore does tons of unnecessary IO (e.g. when .snapshot directories are present, where GPFS makes daily snapshots).

Adding an option to exclude certain directories from what Keep considers would be a good solution.


Subtasks 1 (0 open1 closed)

Task #16487: Review 16425-skip-unused-dirsResolvedWard Vandewege06/12/2020

Actions
Actions #1

Updated by Ward Vandewege almost 4 years ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege almost 4 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Clegg almost 4 years ago

When indexing, it looks like we're already skipping dirs that don't conform to keepstore's storage layout: source:services/keepstore/unix_volume.go#L355

var blockDirRe = regexp.MustCompile(`^[0-9a-f]+$`)
var blockFileRe = regexp.MustCompile(`^[0-9a-f]{32}$`)
...
                if !blockDirRe.MatchString(names[0]) {
                        continue
                }
                ...
                blockdir, err := v.os.Open(blockdirpath)

However, the "empty trash" goroutine does walk the entire tree.

        err := filepath.Walk(v.Root, func(path string, info os.FileInfo, err error) error {
                if err != nil {
                        v.logger.WithError(err).Errorf("EmptyTrash: filepath.Walk(%q) failed", path)
                        return nil
                }
                todo <- dirent{path, info}
                return nil
        })

That walk func should check dirnames and return filepath.SkipDir when appropriate.

Actions #4

Updated by Peter Amstutz over 3 years ago

  • Target version changed from To Be Groomed to 2020-06-17 Sprint
  • Assigned To set to Tom Clegg
Actions #6

Updated by Tom Clegg over 3 years ago

  • Status changed from New to In Progress
Actions #7

Updated by Ward Vandewege over 3 years ago

Tom Clegg wrote:

16425-skip-unused-dirs @ b257b5e4fca36172a79db2fd2cf8ab843cf50d1c -- developer-run-tests: #1915

LGTM, thanks!

Actions #8

Updated by Anonymous over 3 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #9

Updated by Peter Amstutz over 3 years ago

  • Release set to 25
Actions

Also available in: Atom PDF