Stress tests and benchmarks » History » Version 1
Tom Clegg, 03/28/2025 03:14 PM
| 1 | 1 | Tom Clegg | h1. Stress tests and benchmarks |
|---|---|---|---|
| 2 | |||
| 3 | "Stress tests and benchmarks" is a category of tests that we want to maintain and run regularly, but are wasteful to include in every single test run. Examples: |
||
| 4 | * time/memory usage benchmarks ("large manifest" in railsapi) |
||
| 5 | * tests that try to expose race bugs by brute force ("concurrent writes" in keep-web) |
||
| 6 | * tests that report metrics/statistics about algorithm performance ("block repacking" in sdk/go/arvados) |
||
| 7 | |||
| 8 | *Currently,* we deal with this by doing various combinations of |
||
| 9 | * commenting out code |
||
| 10 | * leaving a smaller/faster/less-useful version in the test suite, so we can manually edit the file to do more iterations |
||
| 11 | * skipping tests with a comment to un-skip by editing the source code or setting an environment variable |
||
| 12 | |||
| 13 | As a result the tests are nearly useless in practice. |
||
| 14 | |||
| 15 | *Desired improvement:* a way to add such tests/benchmarks (in any language/component) such that they run |
||
| 16 | * on jenkins weekly |
||
| 17 | * on jenkins before each release |
||
| 18 | * on jenkins on demand (like developer-run-tests) |
||
| 19 | * on a dev machine |
||
| 20 | * ...without bogging down everyday jenkins/dev test jobs. |