Arvados

What's "prefetch 1" and "prefetch 2" ?

Also I'm a little confused how to read this, if the lines in a box-and-whisker plot are supposed to be minimum and maximum how do you have lines that don't touch the box?

This charts make more sense than the old ones, but can you verify what the boxes/whiskers represent here? Ar the whiskers the full range and the box is the 25th and 75th percentiles? Where's the mean and median?

Is the 1800x1MB test is fetching 1800 blocks or fewer blocks with sequential packing?

What is the disk cache setting?

Also this is download speed per file (average) or time to download all the files overall?

Tentatively, it looks like 1 block prefetch may be slightly slower than no prefetch, but also has less variance. However I think it also depends on how many trials you did and exactly what this is measuring. I think we need to go deep into the numbers here and make sure we understand exactly what is happening.

I just wrote a bunch of comments and it ate them when I hit save...

What are the instance types of keep-web and the shell node where you are doing the downloading? We should probably have them both be something like m5n.large.

How many trials are you running? When you showed the data the other day, you had 5 data points. To increase confidence we should run like 20+ trials.

For the small files, a couple of thoughts:

• If I'm reading this right it is running about 1/2 to 1/3 the single file transfer rate, that makes me suspect it is being dominated by connection setup and/or TCP slow start. I'd be curious what the difference is if the same sequential download was done in a single process using a single HTTP session.
• I would like to see a test where the manifest has 1 block per file. To me, the goal of small file prefetch is to improve performance for manifests that are not packed -- so we should have numbers about how it performs in that case.

I'm also curious if less cache (600 MB instead of 1200 MB) makes any difference. Presumably it shouldn't but I think it would be a useful number.

We should also run trials where there is enough cache (2+ GB) so that we have an idea of how it performs in the best case.

For comparison: a single 969M file + warm cache = 530 to 630 MB/s.

Tom Clegg wrote in #note-44:

using 16x concurrent curl processes (xargs -P 16)

16 concurrent curl processes could be stepping on each other. What if the client is reading each file in sequence?

What order is it reading files? Is it favorable for prefetch or counter productive?

Can we do a trial where we shuffle the access order randomly?

Also, I'd still like to see a version of this that uses a single TCP session to see if that meaningfully minimizes overhead from connection setup and TCP slow start.

I'm all for measuring things, but aren't all these performance numbers necessarily affected by disk and network performance? Which will vary across installs and applications? I'm a little wary of overoptimizing our general strategy based on the performance numbers from one specific setup.

What is the algorithm difference between "streaming+prefetch" and "streaming+easyprefetch" ?

If prefetch seems to be a loser, maybe we shouldn't do it at all? Have we found any cases where it clearly beats simple streaming?

Peter Amstutz wrote in #note-51:

What is the algorithm difference between "streaming+prefetch" and "streaming+easyprefetch" ?

streaming+easyprefetch prefetches the next blocks in the current file until it is 64 MiB ahead.

streaming+prefetch prefetches the next blocks in the current file, then the next blocks in the lexically-next file(s) in the directory, until it is 64 MiB ahead.

streaming+prefetch-Nseg prefetches the next blocks in the current file, then up to N blocks in the lexically-next file(s) in the directory, until it is 64 MiB ahead.

If prefetch seems to be a loser, maybe we shouldn't do it at all? Have we found any cases where it clearly beats simple streaming?

#note-42 suggests prefetch might increase the maximum download speed for large files.

Generally, prefetch helps when backend latency is high, but it seems like backend latency is not in fact high now that keepstore itself doesn't add a store-and-forward delay.

Perhaps a better feature is a configurable-size output buffer in keep-web, so (provided the backend throughput is faster than the client throughput, which is also the only situation in which prefetch can help) the client-side buffer drains while the backend is waiting for the next block. Besides using up some more memory, I don't think this would be worse than the current behavior in any situation.

This doesn't help small files at all, but neither does prefetch, so....