Stress tests and benchmarks¶

"Stress tests and benchmarks" is a category of tests that we want to maintain and run regularly, but are wasteful to include in every single test run. Examples:

• time/memory usage benchmarks ("large manifest" in railsapi)
• tests that try to expose race bugs by brute force ("concurrent writes" in keep-web)
• tests that report metrics/statistics about algorithm performance ("block repacking" in sdk/go/arvados)

Currently, we deal with this by doing various combinations of

• commenting out code
• leaving a smaller/faster/less-useful version in the test suite, so we can manually edit the file to do more iterations
• skipping tests with a comment to un-skip by editing the source code or setting an environment variable

As a result the tests are nearly useless in practice.

Desired improvement: a way to add such tests/benchmarks (in any language/component) such that they run

• on jenkins weekly
• on jenkins before each release
• on jenkins on demand (like developer-run-tests)
• on a dev machine
• ...without bogging down everyday jenkins/dev test jobs.