Project

General

Profile

Actions

Feature #9017

open

[Tests] Make slow tests optional (e.g., run-tests.sh --short)

Added by Tom Clegg over 8 years ago. Updated 7 months ago.

Status:
In Progress
Priority:
Normal
Assigned To:
Category:
Tests
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

Some test suites have a few slow tests which contribute disproportionately to the development cycle ("make sure all tests pass before pushing to master" and "make a few changes, run tests to make sure things are still OK").

As a developer, first I want to know as quickly as possible about all of the fast tests that fail. When all of those pass, it makes sense to run the slow tests.

Proposed implementation

Identify slow tests, and skip them when running build/run-tests.sh --short
  • Python tests can use a @slow_test decorator enabled via setup.py test --short-tests-only flag (see 9017-skip-slow-tests @ a39a197)
  • Go packages can use "go test -short". Slow tests in Go packages like keep-web's "download a 1GB file" should already do check testing.Short() and skip or run a faster version.

Jenkins should run the full suite (without --short) at least periodically (e.g., the nightly build).

Jenkins should allow dev jobs to specify whether they want --short.

(Perhaps?) Jenkins' "test when master changes" job should run with --short.


Subtasks 5 (2 open3 closed)

Task #9102: Skip collection performance tests in API suiteResolvedTom Clegg04/26/2016Actions
Task #9101: Communicate --short flag to Rails test suitesResolvedTom Clegg04/26/2016Actions
Task #9052: Review 9017-skip-slow-testsResolvedNico César04/26/2016Actions
Task #9103: Skip collection performance tests in Python SDK suiteIn ProgressTom Clegg04/26/2016Actions
Task #9158: Skip expensive setup/teardown, tooNewTom Clegg04/26/2016Actions
Actions #1

Updated by Tom Clegg over 8 years ago

  • Description updated (diff)
Actions #2

Updated by Tom Clegg over 8 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Clegg over 8 years ago

  • Description updated (diff)
Actions #4

Updated by Tom Clegg over 8 years ago

  • Subject changed from [Tests] Make slow tests optional (e.g., ENABLE_SLOW_TESTS env var) to [Tests] Make slow tests optional (e.g., run-tests.sh --short)
  • Status changed from New to In Progress
Actions #5

Updated by Tom Clegg over 8 years ago

  • Target version set to 2016-04-27 sprint
Actions #6

Updated by Tom Clegg over 8 years ago

  • Category set to Tests
  • Assigned To set to Tom Clegg
Actions #7

Updated by Tom Clegg over 8 years ago

9017-skip-slow-tests @ c9b4764

Actions #8

Updated by Nico César over 8 years ago

reviewed c9b47645638c848bfbc27d21d536995072395106

the change LGTM, and can be merged.

About the the implementation in Jenkins. IMHO the --short should only be triggered manually (i.e. with a parameter of the build) to have a run on the jenkins box and share results quickly. But: even if the test pass it should not upload packages to our repos. which today the job arvados-api-server has a lot of downstream jobs that will end up doing so. One good starting point is to use developer-test-job that will allow this flag to be part of the parameters for quick developer turnaround cycle

Actions #9

Updated by Brett Smith over 8 years ago

  • Target version changed from 2016-04-27 sprint to 2016-05-11 sprint
Actions #10

Updated by Brett Smith over 8 years ago

  • Target version changed from 2016-05-11 sprint to Arvados Future Sprints
  • Story points deleted (0.5)

This is a superbug, where there's basically always more improvement that can be made. Tom will apparently continue to make improvements extracirricularly. But there's not a ton of point in having it on a sprint, since it'll be a long time before we finally agree it's "done."

Actions #11

Updated by Ward Vandewege over 3 years ago

  • Target version deleted (Arvados Future Sprints)
Actions #12

Updated by Peter Amstutz over 1 year ago

  • Release set to 60
Actions #13

Updated by Peter Amstutz 7 months ago

  • Target version set to Future
Actions

Also available in: Atom PDF