Arvados

Have started to look at this. I'm not sure how best to pass through the desire to get an unsigned manifest.

I could add a query param, but then there needs to be some plumbing to make that available in the model where the signed_manifest_text method lives. It feels a bit wrong to do it that way given the separation between the model and controller. Additionally, using a query param would mean that existing clients (like the `arv` cli) would not know anything about the parameter and thus testing would be difficult.

An alternative would be to make another virtual column (I've already implemented this way calling the column `unsigned_manifest_text`), and to have it available for selection (i.e. using `arv collection list -s '["unsigned_manifest_text","uuid"]'`) but not included in the default select list. This does work, and I have successfully set the API server to not return unsigned_manifest_text unless it is explicitly requested by the client.

The issue with that is that the python SDK (or perhaps CLI) seems to set the select list explicitly even if the user does not specify `-s` (i.e. the client seems to be checking the discovery document to figure out what attributes are available and selects all of them, meaning it doesn't really honor the API server's default attribute list).

Therefore, if I add my "unsigned_manifest_list" attribute, a call to `arv collection list` without `-s` ends up requesting both manifest_text and unsigned_manifest_text and gets both of them, which is a waste since the second one is extraneous.

Some options to fix this include:
- use a query param instead (in which case I'd appreciate some guidance on how to plumb that through without breaking the model-controller interface) - perhaps the signing operation belongs in the controller instead of embedded in the model?
- fix the CLI so that it does not take it upon itself to set the select list if the user doesn't specify `-s`
- in the collections controller, enforce that only one of manifest_text or unsigned_manifest_text can ever be included in the @select list (if you ask for both, just return manifest_text).

I've now completed implementing a `count` param (default true) in the API server (for the list method against all resource types) which if set to false will not perform the COUNT queries against the database and should offer performance improvements.

For example: `arv collection list --no-count` will return a list of collections without `items_available` included in the output and without the overhead of the database COUNT.

Applied in changeset arvados|commit:da0aee751d8cb039c9b6b85a03e7d62cb973e3b3.

Using the arv cli, I timed the performance of collection retrieval on our system using a query very similar to what keep-balance uses to compare the performance of the new options. I ran 10 trials each of the 4 permutations of the two options for count ("none" and "exact") and manifest ("manifest_text" vs "unsigned_manifest_text") using the following loop:

$ for i in $(seq 1 10); do for count in exact none; do for manifest in manifest_text unsigned_manifest_text; do echo "${i} ${count} ${manifest}" && time arv collection list --count ${count} -l 2500 -s '["uuid","'${manifest}'","modified_at","portable_data_hash","replication_desired"]' --order '["modified_at","uuid"]' > /dev/null; done; done; done

The results are in the attached plot and are summarised here.

count=="exact"
Min. 1st Qu. Median Mean 3rd Qu. Max.
32.2 35.3 42.8 42.4 49.5 55.3

count=="none"
Min. 1st Qu. Median Mean 3rd Qu. Max.
23.1 25.5 32.6 34.3 40.6 57.9

signed manifest:
Min. 1st Qu. Median Mean 3rd Qu. Max.
36.5 40.7 46.4 46.2 50.1 57.9

unsigned manifest:
Min. 1st Qu. Median Mean 3rd Qu. Max.
23.1 25.5 30.4 30.5 35.1 40.1

count="exact" and signed manifest:
Min. 1st Qu. Median Mean 3rd Qu. Max.
45.5 47.6 49.5 49.3 50.4 55.3

count=="none" and signed manifest:
Min. 1st Qu. Median Mean 3rd Qu. Max.
36.5 38.9 40.7 43.0 43.0 57.9

count=="exact" and unsigned manifest:
Min. 1st Qu. Median Mean 3rd Qu. Max.
32.2 33.2 35.2 35.5 37.3 40.1

count=="none" and unsigned manifest:
Min. 1st Qu. Median Mean 3rd Qu. Max.
23.1 25.2 25.5 25.6 26.4 28.7

Based on the median times, the change from count=="exact" to count=="none" is responsible for the queries being ~24% faster, while the change from signed to unsigned manifests makes the query ~34% faster. Overall, going from count=="exact" with signed manifests to count=="none" with unsigned manifests results in a ~48% faster query.

I'm curious about the outliers in the count=="none" with signed manifest timings, in which sometimes those queries took a very long time (at the extreme, longer than the longest query with count=="exact"!). That could possibly be interference during the tests, although that there are two such outliers in that category and none to speak of in any of the other three categories.

In terms of real-world performance, our keep-balance has gone from taking 11h40m to retrieve all collections before the update to taking 6h30m to retrieve them after the update, which is ~44% faster.

Meanwhile, I continue to be able to dump ALL collection data directly from the database (usnig psql) in just over 3m, which is 99.2% (130x) faster than the current time keep-balance takes to query via the API server (6h30m). This is an improvement from before, in which querying the database directly was 99.6% (233x) faster, but there clearly remains a massive amount of overhead related to the API server.

$ time echo "COPY (select * from collections) TO STDOUT (format text)" | psql -U arvados -w -h localhost arvados_production > /dev/null

real 3m5.614s
user 0m39.836s
sys 0m10.406s

To be fair, if also writing it to disk this ends up taking 50% longer (although keep-balance is loading the data into memory so that should not be an issue there):

$ time echo "COPY (select * from collections) TO STDOUT (format text)" | psql -U arvados -w -h localhost arvados_production > /data/tmp/collections.dump

real 5m29.001s
user 0m41.026s
sys 0m36.107s

We now have 8.3m collections:
$ wc -l /data/tmp/collections.dump
8324815 /data/tmp/collections.dump

Which in text format takes 18GB to represent:
$ ls lh /data/tmp/collections.dump
-rw-r--r- 1 root root 18G Feb 17 12:11 /data/tmp/collections.dump

I'm guessing the remaining overhead probably comes down to ActiveRecord building and processing 8.3m objects for each of these collections?

I wouldn't advocate actually going around the API server for this (i.e. keep-balance should not talk directly to the database), but I'm starting to think it might be necessary to implement some sort of bulk transfer capability in the API so that tools like keep-balance that need all of something can simply ask for a full export of the data, and the API server could skip any unnecessary filtering or framing of the data and just export it directly. Perhaps a new API micro-service could be written in Go to accomplish that?