Feature #12483


[keep-web] writable webdav

Added by Ward Vandewege over 6 years ago. Updated about 6 years ago.

Assigned To:
Target version:
Story points:


The filesystem returned by (*arvados.Collection)FileSystem() needs Create() and OpenFile() methods.
  • the file object returned by Open/Create/OpenFile should be an io.Writer (in addition to io.Seeker and io.ReadCloser as it is now)
  • the filesystem object returned by FileSystem() should have a Collection() method that returns a new arvados.Collection with the same UUID as the original collection's UUID, and a manifest text and PDH that reflect the modified filesystem.
  • modifying the filesystem (by opening/writing files) does not modify the original collection at all.
keep-web will require some changes:
  • webdav code should call OpenFile with the requested flag argument, and pass through Write calls, instead of calling Open and stubbing Write.)
  • the collection uuid→pdh cache needs to be sufficiently write-aware that a sequence of writes (by a single client) behaves predictably
  • the collection cache needs to be sufficiently write-aware that writing to a collection and then reading from it using its old PDH does not return the modified data
This story does not require (and should not be held up by):
  • De-duplicating existing code that writes to Keep, like crunch-run
  • Optimizing generalized write performance (block packing for small files, performance when doing many short writes to many files at once)
  • Optimizing keep-web performance when writing a large file using many small writes in separate webdav requests
  • Avoiding lost updates when saving updated collections
This story does require:
  • Good block packing in the most common/easy cases (sequential short writes to a single large file result in 64 MiB blocks)
  • Correct behavior for arbitrary sequences of read/write/seek on multiple files in a single filesystem
  • Correct behavior when multiple goroutines are concurrently updating one or more files in a filesystem when each goroutine has its own file object (however, callers are responsible for their own goroutine safety when sharing a single file object)

Subtasks 1 (0 open1 closed)

Task #12578: Review 12483-writable-fsResolvedTom Clegg10/25/2017Actions

Related issues

Related to Arvados - Idea #11876: [R SDK] Create a Bioconductor/R SDKClosedFuad Muhic06/20/2017Actions

Also available in: Atom PDF