Bug #12715
closed
[keep-web] Slow PROPFIND
Description
This is very fast:
curl -v -X GET -H "Authorization: OAuth2 4invqy35tf70t7hmvdc83ges8ug9cklhgqq1l8gj2cjn18teuq" https://collections.4xphq.arvadosapi.com/c=4xphq-4zz18-9d5b0qm4fgijeyi/_/
This is very slow:
curl -v -X PROPFIND -H "Authorization: OAuth2 4invqy35tf70t7hmvdc83ges8ug9cklhgqq1l8gj2cjn18teuq" https://collections.4xphq.arvadosapi.com/c=4xphq-4zz18-9d5b0qm4fgijeyi/_/
The behavior seems to be any PROPFIND response bigger than (some buffer size) gets reported in chunks, with distinct sometimes long pauses between each chunk.
I noticed that keep-web is using Transfer-Encoding: chunked in both cases.
Would this behave better if it generated the index in a buffer and reported accurate content-length?
Updated by Peter Amstutz over 6 years ago
On #12714 we found that PROPFIND is still horribly slow (but GET is fine) even when bypassing nginix, so that strongly indicates the bug is in keep-web.
Updated by Tom Morris over 6 years ago
- Target version changed from 2017-12-06 Sprint to 2017-12-20 Sprint
Updated by Tom Clegg over 6 years ago
This might be related to chunked transfer encoding, but even so, I don't think it means chunked transfer encoding is unusable.
It's possible the output is simply slow, and chunked transfer encoding makes it look chunky.
Updated by Tom Clegg over 6 years ago
webdav is reading the first 512 bytes of each file if it doesn't recognize the extension.
https://github.com/golang/net/blob/master/webdav/prop.go#L393-L395
This often involves getting a 64 MiB block from Keep.
We might work around this by passing dummy files (returning io.ErrUnexpectedEOF on every read) when the request method is PROPFIND.
Updated by Tom Clegg over 6 years ago
12715-slow-propfind @ 737f7d618ffc18152c5d09416884910bd180d890
Updated by Peter Amstutz over 6 years ago
Tom Clegg wrote:
12715-slow-propfind @ 737f7d618ffc18152c5d09416884910bd180d890
Calling the flag webdavFS.reading seems a little confusing, perhaps flipping the logic and calling it "propfind" would be clearer?
Updated by Tom Clegg over 6 years ago
Ha. I started with a true-means-fail flag but then changed it to be more consistent with "writing". Sticking with describing what it does, rather than why -- how about alwaysReadEOF?
12715-slow-propfind @ 7aa1bc6eaf4d340d6e45226003b776d521309d69
Updated by Anonymous over 6 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|5512f001241c17152087ed5b5a28f94a9c629541.