Arvados

Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK").

As a developer, first I want to know as quickly as possible about all of the fast tests that fail. When all of those pass, it makes sense to run the slow tests.

Proposed implementation¶

Identify slow tests, and skip them when running build/run-tests.sh --short

• Python tests can use a @slow_test decorator enabled via setup.py test --short-tests-only flag (see 9017-skip-slow-tests @ a39a197)
• Go packages can use "go test -short". Slow tests in Go packages like keep-web's "download a 1GB file" should already do check testing.Short() and skip or run a faster version.

Jenkins should run the full suite (without --short) at least periodically (e.g., the nightly build).

Jenkins should allow dev jobs to specify whether they want --short.

(Perhaps?) Jenkins' "test when master changes" job should run with --short.