Feature #9017

Updated by Tom Clegg over 5 years ago

Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK").

As a developer, first I want to know as quickly as possible about all of the _fast_ tests that fail. When all of those pass, it makes sense to run the slow tests.

h2. Proposed implementation

Identify slow tests, and skip them unless the ENABLE_SLOW_TESTS environment variable is set.
* Python tests can use a @@slow_test@ decorator (see 9017-skip-slow-tests @ commit:a58cc88)
* run-tests.sh should invoke "go test -short" if ENABLE_SLOW_TESTS is unset. Slow tests in Go packages like keep-web's "download a 1GB file" should check @testing.Short()@ (rather than checking ENABLE_SLOW_TESTS directly) and skip or run a faster version.

Jenkins should run with ENABLE_SLOW_TESTS at least periodically (e.g., the nightly build).

Jenkins should allow dev jobs to specify whether they want ENABLE_SLOW_TESTS.

(Perhaps?) Jenkins' "test when master changes" job should run without ENABLE_SLOW_TESTS.