Project

General

Profile

Feature #9017

Updated by Tom Clegg almost 8 years ago

Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK"). 

 As a developer, first I want to know as quickly as possible about all of the _fast_ tests that fail. When all of those pass, it makes sense to run the slow tests. 

 h2. Proposed implementation 

 Identify slow tests, and skip them when running @build/run-tests.sh --short@ unless the ENABLE_SLOW_TESTS environment variable is set. 
 * Python tests can use a @@slow_test@ decorator enabled via ENABLE_SLOW_TESTS env var (see 9017-skip-slow-tests @ commit:a58cc88) ...or an argument to setup.py? 
 * Go packages can use run-tests.sh should invoke "go test -short". -short" if ENABLE_SLOW_TESTS is unset.    Slow tests in Go packages like keep-web's "download a 1GB file" should check @testing.Short()@ (rather than checking ENABLE_SLOW_TESTS directly) and skip or run a faster version. 

 Jenkins should run the full suite (without @--short@) with ENABLE_SLOW_TESTS at least periodically (e.g., the nightly build). 

 Jenkins should allow dev jobs to specify whether they want @--short@. ENABLE_SLOW_TESTS. 

 (Perhaps?) Jenkins' "test when master changes" job should run with @--short@. without ENABLE_SLOW_TESTS. 

Back