Feature #9017

Updated by Tom Clegg over 5 years ago

Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK").

As a developer, first I want to know as quickly as possible about all of the _fast_ tests that fail. When all of those pass, it makes sense to run the slow tests.

h2. Proposed implementation

Identify slow tests, and skip them when running @build/run-tests.sh --short@ unless the ENABLE_SLOW_TESTS environment variable is set.
* Python tests can use a @@slow_test@ decorator enabled via ENABLE_SLOW_TESTS env var (see 9017-skip-slow-tests @ commit:a58cc88) ...or an argument to setup.py?
* Go packages can use run-tests.sh should invoke "go test -short". -short" if ENABLE_SLOW_TESTS is unset. Slow tests in Go packages like keep-web's "download a 1GB file" should check @testing.Short()@ (rather than checking ENABLE_SLOW_TESTS directly) and skip or run a faster version.

Jenkins should run the full suite (without @--short@) with ENABLE_SLOW_TESTS at least periodically (e.g., the nightly build).

Jenkins should allow dev jobs to specify whether they want @--short@. ENABLE_SLOW_TESTS.

(Perhaps?) Jenkins' "test when master changes" job should run with @--short@. without ENABLE_SLOW_TESTS.