Project

General

Profile

Bug #5901

Updated by Tom Morris about 7 years ago

Attached are I've attached two files.    The first is a simple Python script that uses threads to fetch the same collection object from the API server multiple times simultaneously.    Currently, the collection's manifest is 75492690 43924557 bytes.    The collection UUID is su92l-4zz18-wd2va9q9lnfx6ga 

 The log file was generated by running: 

 <pre><code class="shell">for n in 2 4 6 8; do python multi.py "$n" || break; done | tee multi.log</code></pre> 

 Simply, it shows that performance takes a noticeable dive as the number of simultaneous requests increase.    The eight-thread calls never succeed; instead they raised a timeout exception.    This problem just bit a real user: parallelizing over many files in this collection, the first batch of parallel tasks all failed because they all tried to fetch the collection simultaneously, and timed out waiting for an API server response.    We have to improve performance here to make sure this use pattern doesn't fail.

Back