Arvados

I haven't been able to reproduce a hung keepproxy test myself since a221be676f0650993e76a9e402b93fa4ef7b5cf6 but jenkins has done it twice, in "remainder" builds 1933 and 1949.

While investigating other "remainder" failures I fixed a keepstore test bug that broke builds 1932 and 1945: 6c56f80642aae0ec0ff0bfc939ae33dd49e0ca6b

While attempting to reproduce the keepproxy hang locally, I did hit occasional "concurrent map writes" panics, which seem to have been caused by inadvertently copying a sync.Mutex after first use. After fixing that in bce4ee3aabf1dabc41ea5b231f0407cad3347f11, keepproxy tests seem reliable here (tried 64x, no failures).

It's possible that same mutex bug could cause tests to hang -- e.g., if the mutex was copied while locked -- and that the 10 minute test timeout isn't effective when it happens because https://github.com/golang/go/issues/24050 .

14 "run-tests-remainder" runs since merging this, with no timeouts or failures.