Project

General

Profile

Actions

Idea #13110

open

[WebDAV] WebDAV performance enhancements

Added by Tom Morris about 6 years ago. Updated 27 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

From https://dev.arvados.org/issues/12483#note-38

  • Cache filesystem objects in keep-web, instead of rebuilding from manifest on each request → less time, more RAM
  • Reduce memory use (e.g., pass keepclient pointer into storedSegment methods instead of keeping a copy in the struct)
  • Incremental loading (don't populate a dirnode until someone accesses it)
  • Cache/evict partial data blocks
  • Test/benchmark using a variety of webdav clients (Windows, linux davfs, OSX?)

From #13272:

Cache the filesystem layout, so a sequence of N webdav requests with the same token and same/overlapping paths don't need to retrieve the whole filesystem layout from API server N times
Use a single cache entry for any given {collection, token} tuple, regardless of whether it's accessed via /c=UUID/, /by_id/UUID/, /by_id/PDH/, /users/alice/foo/, etc.


Related issues

Related to Arvados - Feature #13272: [keep-web] read/write support for /by_id and /usersNewActions
Related to Arvados Epics - Idea #18342: Keep performance optimizationNew08/01/202303/31/2024Actions
Actions #1

Updated by Tom Morris about 6 years ago

  • Description updated (diff)
Actions #2

Updated by Tom Morris about 6 years ago

  • Target version set to To Be Groomed
Actions #3

Updated by Tom Morris about 6 years ago

  • Description updated (diff)
Actions #4

Updated by Tom Morris about 5 years ago

  • Description updated (diff)
Actions #5

Updated by Tom Morris about 5 years ago

  • Related to Feature #13272: [keep-web] read/write support for /by_id and /users added
Actions #6

Updated by Tom Morris about 5 years ago

  • Subject changed from [WebDAV] Cleanup writable WebDAV loose ends to [WebDAV] WebDAV performance enhancements
Actions #7

Updated by Peter Amstutz over 2 years ago

  • Target version deleted (To Be Groomed)
Actions #8

Updated by Peter Amstutz about 1 year ago

  • Release set to 60
Actions #10

Updated by Peter Amstutz 6 months ago

  • Related to Idea #18342: Keep performance optimization added
Actions #11

Updated by Peter Amstutz 27 days ago

  • Target version set to Future
Actions

Also available in: Atom PDF