Arvados

• changes logging format to JSON (was timestamped plaintext)
• adds backend storage metrics: keep_{garbage,lost,overreplicated,total,transient,underreplicated}_{blocks,bytes,replicas}
• adds keep-balance timing metrics: keepbalance_{changeset_compute,get_state,send_pull_list,send_trash_lists,sweep}_seconds
• updates doc page

TODO: require management token

Should the ‘Listen’ config param be added to usage.go? The rest lgtm.

Metrics endpoints for keepstore, keep-web, keep-balance now require ManagementToken, if one is configured.

Added Listen and ManagementToken to usage.go.

14285-keep-balance-metrics @ 1f7a87ea4c342d16e0992872e2893cb6a2da92e9 https://ci.curoverse.com/view/Developer/job/developer-run-tests/928/

14285-keep-balance-metrics @ 2e50177344ad50dc5d6c599445c272e200636af8 https://ci.curoverse.com/view/Developer/job/developer-run-tests/929/

Fixes failing sdk/go/health test.

A couple of questions:

• I’m not understanding how LoadToken() comes into play with the rest of the code, it seems to not being called from anywhere, could you please explain?
• Is sdk/go/auth getting big enough to deserve some testing?

Lucas Di Pentima wrote:

A couple of questions:

• I’m not understanding how LoadToken() comes into play with the rest of the code, it seems to not being called from anywhere, could you please explain?

It's a middleware that attaches a *Credentials to the request, for CredentialsFromRequest can return when called (perhaps repeatedly) by inner handlers. The most likely use for it is in controller, but not until it gets cleaned up a bit.

• Is sdk/go/auth getting big enough to deserve some testing?

Yes, added some handler tests. There's room for more, but how far should we go?

14285-keep-balance-metrics @ 7ad5d87a9e2d67224ed440c9320f1850cbaf6ae1

Thanks for the explanation, I had the suspicion that it was for future use.
This LGTM, please merge.

• adds arvados_keep_dedup_block_ratio and ..._byte_ratio metrics
• rough measure of storage cost savings as a result of content addressing
• ratio=2.0 would mean the average block is referenced by two collections

Just one question: Would it be convenient to use uin64 instead of int64 on collectionB.+s vars?
The rest LGTM.

Lucas Di Pentima wrote:

Just one question: Would it be convenient to use uin64 instead of int64 on collectionB.+s vars?

I'm inclined to use signed ints anywhere arithmetic might be done -- uints have weird properties like 0-1 > 0 that can produce subtle bugs later on.