Bug #12715

[keep-web] Slow PROPFIND

Added by Peter Amstutz about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
12/15/2017
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

This is very fast:

curl -v -X GET -H "Authorization: OAuth2 4invqy35tf70t7hmvdc83ges8ug9cklhgqq1l8gj2cjn18teuq" https://collections.4xphq.arvadosapi.com/c=4xphq-4zz18-9d5b0qm4fgijeyi/_/

This is very slow:

curl -v -X PROPFIND -H "Authorization: OAuth2 4invqy35tf70t7hmvdc83ges8ug9cklhgqq1l8gj2cjn18teuq" https://collections.4xphq.arvadosapi.com/c=4xphq-4zz18-9d5b0qm4fgijeyi/_/

The behavior seems to be any PROPFIND response bigger than (some buffer size) gets reported in chunks, with distinct sometimes long pauses between each chunk.

I noticed that keep-web is using Transfer-Encoding: chunked in both cases.

Would this behave better if it generated the index in a buffer and reported accurate content-length?


Subtasks

Task #12769: Review 12715-slow-propfindClosedPeter Amstutz

Associated revisions

Revision 5512f001
Added by Tom Clegg almost 2 years ago

Merge branch '12715-slow-propfind'

fixes #12715

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#2 Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)

#3 Updated by Peter Amstutz about 2 years ago

On #12714 we found that PROPFIND is still horribly slow (but GET is fine) even when bypassing nginix, so that strongly indicates the bug is in keep-web.

#4 Updated by Tom Morris about 2 years ago

  • Target version changed from 2017-12-06 Sprint to 2017-12-20 Sprint

#5 Updated by Tom Clegg about 2 years ago

This might be related to chunked transfer encoding, but even so, I don't think it means chunked transfer encoding is unusable.

It's possible the output is simply slow, and chunked transfer encoding makes it look chunky.

#6 Updated by Tom Clegg about 2 years ago

  • Assigned To set to Tom Clegg

#7 Updated by Tom Clegg almost 2 years ago

webdav is reading the first 512 bytes of each file if it doesn't recognize the extension.

https://github.com/golang/net/blob/master/webdav/prop.go#L393-L395

This often involves getting a 64 MiB block from Keep.

We might work around this by passing dummy files (returning io.ErrUnexpectedEOF on every read) when the request method is PROPFIND.

#8 Updated by Tom Clegg almost 2 years ago

  • Status changed from New to In Progress

#10 Updated by Peter Amstutz almost 2 years ago

Tom Clegg wrote:

12715-slow-propfind @ 737f7d618ffc18152c5d09416884910bd180d890

Calling the flag webdavFS.reading seems a little confusing, perhaps flipping the logic and calling it "propfind" would be clearer?

#11 Updated by Tom Clegg almost 2 years ago

Ha. I started with a true-means-fail flag but then changed it to be more consistent with "writing". Sticking with describing what it does, rather than why -- how about alwaysReadEOF?

12715-slow-propfind @ 7aa1bc6eaf4d340d6e45226003b776d521309d69

#12 Updated by Anonymous almost 2 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF