Feature #9017
open
[Tests] Make slow tests optional (e.g., run-tests.sh --short)
Description
Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK").
As a developer, first I want to know as quickly as possible about all of the fast tests that fail. When all of those pass, it makes sense to run the slow tests.
Proposed implementation¶
Identify slow tests, and skip them when runningbuild/run-tests.sh --short
- Python tests can use a
@slow_testdecorator enabled viasetup.py test --short-tests-onlyflag (see 9017-skip-slow-tests @ a39a197) - Go packages can use
"go test -short". Slow tests in Go packages like keep-web's "download a 1GB file"shouldalready do checktesting.Short()and skip or run a faster version.
Jenkins should run the full suite (without --short) at least periodically (e.g., the nightly build).
Jenkins should allow dev jobs to specify whether they want --short.
(Perhaps?) Jenkins' "test when master changes" job should run with --short.
Updated by Tom Clegg over 8 years ago
- Subject changed from [Tests] Make slow tests optional (e.g., ENABLE_SLOW_TESTS env var) to [Tests] Make slow tests optional (e.g., run-tests.sh --short)
- Status changed from New to In Progress
Updated by Tom Clegg over 8 years ago
- Category set to Tests
- Assigned To set to Tom Clegg
Updated by Nico César over 8 years ago
reviewed c9b47645638c848bfbc27d21d536995072395106
the change LGTM, and can be merged.
About the the implementation in Jenkins. IMHO the --short should only be triggered manually (i.e. with a parameter of the build) to have a run on the jenkins box and share results quickly. But: even if the test pass it should not upload packages to our repos. which today the job arvados-api-server has a lot of downstream jobs that will end up doing so. One good starting point is to use developer-test-job that will allow this flag to be part of the parameters for quick developer turnaround cycle
Updated by Brett Smith over 8 years ago
- Target version changed from 2016-04-27 sprint to 2016-05-11 sprint
Updated by Brett Smith over 8 years ago
- Target version changed from 2016-05-11 sprint to Arvados Future Sprints
- Story points deleted (
0.5)
This is a superbug, where there's basically always more improvement that can be made. Tom will apparently continue to make improvements extracirricularly. But there's not a ton of point in having it on a sprint, since it'll be a long time before we finally agree it's "done."
Updated by Ward Vandewege over 3 years ago
- Target version deleted (
Arvados Future Sprints)
Updated by