Feature #9017

[Tests] Make slow tests optional (e.g., run-tests.sh --short)

Added by Tom Clegg almost 3 years ago. Updated almost 3 years ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
Tests
Target version:
Start date:
04/26/2016
Due date:
% Done:

60%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK").

As a developer, first I want to know as quickly as possible about all of the fast tests that fail. When all of those pass, it makes sense to run the slow tests.

Proposed implementation

Identify slow tests, and skip them when running build/run-tests.sh --short
  • Python tests can use a @slow_test decorator enabled via setup.py test --short-tests-only flag (see 9017-skip-slow-tests @ a39a197)
  • Go packages can use "go test -short". Slow tests in Go packages like keep-web's "download a 1GB file" should already do check testing.Short() and skip or run a faster version.

Jenkins should run the full suite (without --short) at least periodically (e.g., the nightly build).

Jenkins should allow dev jobs to specify whether they want --short.

(Perhaps?) Jenkins' "test when master changes" job should run with --short.


Subtasks

Task #9102: Skip collection performance tests in API suiteResolvedTom Clegg

Task #9101: Communicate --short flag to Rails test suitesResolvedTom Clegg

Task #9052: Review 9017-skip-slow-testsResolvedNico César

Task #9103: Skip collection performance tests in Python SDK suiteIn ProgressTom Clegg

Task #9158: Skip expensive setup/teardown, tooNewTom Clegg

Associated revisions

Revision e1de8892
Added by Tom Clegg almost 3 years ago

Merge branch '9017-skip-slow-tests'

refs #9017

Revision 1a39dff7
Added by Tom Clegg almost 3 years ago

Merge branch '9017-apiserver-short-tests'

refs #9017

History

#1 Updated by Tom Clegg almost 3 years ago

  • Description updated (diff)

#2 Updated by Tom Clegg almost 3 years ago

  • Description updated (diff)

#3 Updated by Tom Clegg almost 3 years ago

  • Description updated (diff)

#4 Updated by Tom Clegg almost 3 years ago

  • Subject changed from [Tests] Make slow tests optional (e.g., ENABLE_SLOW_TESTS env var) to [Tests] Make slow tests optional (e.g., run-tests.sh --short)
  • Status changed from New to In Progress

#5 Updated by Tom Clegg almost 3 years ago

  • Target version set to 2016-04-27 sprint

#6 Updated by Tom Clegg almost 3 years ago

  • Category set to Tests
  • Assigned To set to Tom Clegg

#7 Updated by Tom Clegg almost 3 years ago

9017-skip-slow-tests @ c9b4764

#8 Updated by Nico César almost 3 years ago

reviewed c9b47645638c848bfbc27d21d536995072395106

the change LGTM, and can be merged.

About the the implementation in Jenkins. IMHO the --short should only be triggered manually (i.e. with a parameter of the build) to have a run on the jenkins box and share results quickly. But: even if the test pass it should not upload packages to our repos. which today the job arvados-api-server has a lot of downstream jobs that will end up doing so. One good starting point is to use developer-test-job that will allow this flag to be part of the parameters for quick developer turnaround cycle

#9 Updated by Brett Smith almost 3 years ago

  • Target version changed from 2016-04-27 sprint to 2016-05-11 sprint

#10 Updated by Brett Smith almost 3 years ago

  • Target version changed from 2016-05-11 sprint to Arvados Future Sprints
  • Story points deleted (0.5)

This is a superbug, where there's basically always more improvement that can be made. Tom will apparently continue to make improvements extracirricularly. But there's not a ton of point in having it on a sprint, since it'll be a long time before we finally agree it's "done."

Also available in: Atom PDF