[Tests] Make slow tests optional (e.g., run-tests.sh --short)
Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK").
As a developer, first I want to know as quickly as possible about all of the fast tests that fail. When all of those pass, it makes sense to run the slow tests.
Proposed implementation¶Identify slow tests, and skip them when running
- Python tests can use a
@slow_testdecorator enabled via
setup.py test --short-tests-onlyflag (see 9017-skip-slow-tests @ a39a197)
- Go packages can use
"go test -short". Slow tests in Go packages like keep-web's "download a 1GB file"
shouldalready do check
testing.Short()and skip or run a faster version.
Jenkins should run the full suite (without
--short) at least periodically (e.g., the nightly build).
Jenkins should allow dev jobs to specify whether they want
(Perhaps?) Jenkins' "test when master changes" job should run with
#8 Updated by Nico César over 2 years ago
the change LGTM, and can be merged.
About the the implementation in Jenkins. IMHO the --short should only be triggered manually (i.e. with a parameter of the build) to have a run on the jenkins box and share results quickly. But: even if the test pass it should not upload packages to our repos. which today the job arvados-api-server has a lot of downstream jobs that will end up doing so. One good starting point is to use developer-test-job that will allow this flag to be part of the parameters for quick developer turnaround cycle
#10 Updated by Brett Smith over 2 years ago
- Target version changed from 2016-05-11 sprint to Arvados Future Sprints
- Story points deleted (
This is a superbug, where there's basically always more improvement that can be made. Tom will apparently continue to make improvements extracirricularly. But there's not a ton of point in having it on a sprint, since it'll be a long time before we finally agree it's "done."