Arvados

We should also consider providing an S3-compatible API.

Does this include browsing projects? (Probably not, but for the desktop filesystem mount use case, it probably should). Ideally it would provide the same FS view as arv-mount.

• test case using cadaver client (run-tests.sh now needs "apt install cadaver")
• manual testing with mount.davfs works (it prompts for user&pass; user can be anything, pass is api token)

• I gave up trying to make cadaver do http authentication -- I'm guessing the compile-time option to support .netrc is not enabled in the debian package, and it seems to ignore u:p in http://u:p@host:port/
• "find /mnt" is slow on a collection with many directories. It does one http request per directory instead of using depth>1, and keep-web uses lots of CPU. I think it's doing lots of un-optimized manifest wrangling. My plan is to fix this by caching the http.FileSystem instead of just the collection.
• There's no new functionality like browsing available collections. You still need to specify the collection ID in the URL in one of the various supported ways. The only difference is that now a webdav client can get the directory listings that used to be available only to a human or html-scraper.

As far as I can see, this looks good.

I've encountered the cached listing behavior I mentioned on the chat, where a listing gets cached and changes are not reflected. If this is client dependent, maybe it would be safe to force listing cache invalidation to avoid hard to debug issues with webdav clients?

Lucas Di Pentima wrote:

I've encountered the cached listing behavior I mentioned on the chat, where a listing gets cached and changes are not reflected. If this is client dependent, maybe it would be safe to force listing cache invalidation to avoid hard to debug issues with webdav clients?

1. Get directory listing from keep-web → receive version 1
2. Update collection using REST API → current version is 2
3. Get directory listing from keep-web → receive version 1, but expect version 2

(Is this a more general issue, or is there also something I'm missing that makes cached webdav directory listings more confusing than cached file content?)

• listen for cache invalidation events, either from arvados-ws or more directly from postgresql
• option for a separate TTL config for the uuid→pdh cache (could be set to zero, or something else shorter than the pdh->manifest cache TTL)

• allow Authorization header in cross-origin requests (see commit message 337de2e3d)
• fix crash on some dir-listing reqs with no trailing slash
• huge performance improvement in 991d7d796 (webdav does a lot more file-opening than I thought -- before this, we were parsing the whole manifest multiple times for each file returned in a dir listing!)

Latest updates lgtm, lazy file opening is a cool idea!
Regarding cache invalidation, I was seeing something like you describe: being connected with cadaver client, asked a listing, then uploaded something with arv-put and asked a listing again, resulting in the same output. In my opinion, a uuid->pdh TTL config would be enough and simpler to implement than an event handler.

• merged master
• separate TTL for uuid->pdh cache (default 5 seconds)

Updates at ec0c244be178aed7af0cf990a256dda557034b68 LGTM.
Local keep-web tests didn't complain so I suppose we're not testing those TTLs.
Are cache parameters configurable via keep-web.yml, they're part of the config struct but don't know if they get picked up from the file. If they're configurable, maybe we should document the difference between both somewhere.

There's an experiment using browser-side JS to get directory listings:

spike-wb-browse-collection @ 20dbebcdd863589f47bce138418cfcacd5f32b2e