Idea #12308
Updated by Tom Clegg about 4 years ago
Background:
Python+llfuse was expedient and has done lots of good work for us, but it's not promising as a long term (fast+reliable+maintainable) solution.
Implementation:
* collection-backed filesystem from #12483, plus more general arvados-backed filesystem ("by_id" directory, etc, same as the one exported via webdav) from #13111
* present as fuse using a library like https://godoc.org/bazil.org/fuse or https://godoc.org/github.com/billziss-gh/cgofuse/fuse
* package as a subcommand ("mount") of the source:cmd/arvados-client program
TBD:
* Approach for handling websocket "update" events
* Selectable mechanisms/options for syncing to server (fflush, fsync, close) (on a shell node, flush-on-close, flush-periodically, or flush-after-idle-time might be best; in crunch-run, flush-on-exit might be best)
* Desired behavior when updates conflict (write error? clobber? create "oops,clobbered" file?)
Other current bugs/limitations:
* Not command-line compatible with arv-mount
* Logging is not great
* Appears to leak memory when using large collection
* No docs
* No way to control overall cache size (currently collectionfs can use lots of RAM in certain non-sequential write scenarios; we need the ability to trade speed for space efficiency in memory-constrained environments)
* No warnings given when cache is thrashing
* No application level instrumentation (just optional Go pprof)
* Special @.arvados#collection@ file is incomplete (has manifest_text but not uuid, pdh)