Arvados

It appears that Rails "streaming" actually means rendering and delivering the page incrementally, as opposed to the normal mode of operation where the entire page is rendered before being delivered. With that use case in mind, it's not so surprising that it hangs on to the data being sent out. So our current method is probably the entirely wrong way to do file download.

To actually stream data out to the socket, it looks like we need to either use send_file() (and rig something up using arv-mount?), Rails 4 live streaming (include ActionController::Live), write our own Rack middleware to specifically intercept file downloads (I've written a couple of rack middlewares already, it's not that hard), or redirect downloads to some to-be-decided alternate service such as keepproxy.

Somewhat related to this is the need to support range clauses so we can do partial downloads for things like large log files.

3616-live-stream @ 1586924

• Before: ruby process grows from 819m to 1900m, then client starts receiving data.
• After: ruby process grows from 819m to 826m and stays there while I download.

Ok, I've verified this works with puma and passenger. It seems that webrick doesn't support streaming and so it still wants to buffer the whole thing, but that's not a blocker for production

Unfortunately it uses chunked transfer encoding, which turns out to have a few problems:

• it adds a little bit of overhead
• you can't specify content-length so the downloader doesn't know how much data to expect a
• on Firefox these downloads don't show up in the download dialog at all, although they do get downloaded. (on chromium it works) (at least, this is what I'm seeing, its possible my firefox is broken).

Also, I suggest adding --batch-progress or --no-progress the arv-get invocation to avoid cluttering up the logs.

Peter Amstutz wrote:

Ok, I've verified this works with puma and passenger. It seems that webrick doesn't support streaming and so it still wants to buffer the whole thing, but that's not a blocker for production

Right, this is not the first reason not to use Webrick in production. :) And it seems to make Webrick no more broken than it was before this change, so I don't feel bad about anyone who happens to be running Webrick for some reason.

Unfortunately it uses chunked transfer encoding, which turns out to have a few problems:

• it adds a little bit of overhead
• you can't specify content-length so the downloader doesn't know how much data to expect a
• on Firefox these downloads don't show up in the download dialog at all, although they do get downloaded. (on chromium it works) (at least, this is what I'm seeing, its possible my firefox is broken).

Those problems are regrettable, but less regrettable than using O(N) RAM.

Also, I suggest adding --batch-progress or --no-progress the arv-get invocation to avoid cluttering up the logs.

This shouldn't be necessary since arv-get already defaults to --no-progress unless stderr isatty.

Thanks