Arvados

See https://arvados.org/issues/3198#note-39 and https://arvados.org/issues/3198#note-40 for some benchmarking notes around FUSE and Collections.

It may go without saying, but writing tests for write operations is blocked by #3198. However, it should be possible to start the performance test framework with tests for read operations, in a way that's extensible to write operations as well.

Regarding branch 6219-fuse-performance-testing:

• This branch is from 3198-writable-fuse branch

• Includes some tests for testing collection operations
  • Create collection with files with multiple blocks; and move and remove a file. The test takes forever if I use too many files or blocks. So, used 6 files with 2 blocks each
  • Create collection and add 2 streams with 200 files each to it; move and remove one file. Similar to the above test, it takes too long if I push the numbers much higher than this.
  • Create collection and add files using magic dir; move all files into another collection
  • Create collection and add files using magic dir; move one file at a time into another collection

• I could not figure out how to add multiple streams to a collection when testing with magic dir. During review, please provide a hint as to how to do this. Thanks.

Radhika Chippada wrote:

I did not add test to retrieve project contents at this time.

Was there some obstacle that made this impossible or impractical at this time? The performance of this specific operation is one that we've received bug reports about (e.g., #6019), so it's one that we know for sure users have interest in. The same goes for listing files in a large collection (#5662)—I'm not sure if that's implicitly covered in one of your existing tests, but it would be good to have profiling results for that in isolation.

The branch now contains test that lists a project's contents as well.

Radhika Chippada wrote:

The branch now contains test that lists a project's contents as well.

Thanks. And listing the files in a collection with many files?

In magicDirTestMoveFiles_oneEachIntoAnother

The @profiled annotation is around a loop which calls "pool.apply". I don't know how much overhead the Python multiprocessing adds compared to the function being called, so it would be better to have the outer loop and @profiled annotation to be part of magicDirTest_MoveFileFromCollection instead.

Thanks Peter for the comments. I updated the branch to profile smaller code fragments. Please take another look.

I'm not entirely sure this answers the performance questions about FUSE that I would want to ask, but this branch has been outstanding for awhile so let's go ahead and merge.

Peter Amstutz wrote:

I'm not entirely sure this answers the performance questions about FUSE that I would want to ask, but this branch has been outstanding for awhile so let's go ahead and merge.

One of our criteria for branch reviews is, "Does the branch do what the story specifies?" So, does this branch provide profile data for the use cases listed in the description? If so, then we're golden. If not, I'd like to know more about what the discrepancies are and why they exist.

• Using FUSE for a collection with streams: 2, files_per_stream: 200, bytes_per_file: 1

• Using FUSE for a collection with streams: 2, files_per_stream: 3, blocks_per_file: 2 bytes_per_block: 2**26

• Using Magi Dir for a collection with streams: 2, files_per_stream: 200, bytes_per_file: 1