Story #2761

[Workbench] diagnostic suite runs against a production instance

Added by Tom Clegg about 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
-
Target version:
Start date:
09/15/2014
Due date:
% Done:

100%

Estimated time:
(Total: 1.00 h)
Story points:
3.0

Subtasks

Task #3870: Review branch: 2761-diagnostic-suiteResolvedRadhika Chippada

Associated revisions

Revision 2d04e7d3
Added by Radhika Chippada over 6 years ago

closes #2761
Merge branch '2761-diagnostic-suite'

History

#1 Updated by Tom Clegg over 6 years ago

  • Target version set to Arvados Future Sprints

#2 Updated by Tom Clegg over 6 years ago

  • Subject changed from Workbench diagnostic suite runs against a production instance to [Workbench] diagnostic suite runs against a production instance

#3 Updated by Ward Vandewege over 6 years ago

  • Target version changed from Arvados Future Sprints to 2014-09-17 sprint

#4 Updated by Radhika Chippada over 6 years ago

  • Assigned To set to Radhika Chippada

#5 Updated by Radhika Chippada over 6 years ago

  • Status changed from New to In Progress

#6 Updated by Radhika Chippada over 6 years ago

  • Added "apps/workbench/test/diagnostics" test directory
  • Added diagnostics testing configuration to application.default.yml with explanation as to how to configure diagnostics testing
  • Diagnostics testing currently provides ability to configure pipeline running.
  • Added apps/workbench/test/diagnostics/pipeline_test.rb that runs the pipelines configured using the configuration parameter.
  • The diagnostics tests can be executed using a command as follows:
    • RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb (to run just this test file)
    • RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/* (to run all test files in this directory, though currently there is only test file)
  • The diagnostics tests will run against the workbench URL configured in the config.

#7 Updated by Tom Clegg over 6 years ago

I like having the diagnostics tests in a separate directory, but I think we should also separate them from the other RAILS_ENV environments:

In apps/workbench/config/application.default.yml
  • Add a new section called "diagnostics" with the examples enabled (leave them as false in the common section) - this should let us run the diagnostics using RAILS_ENV=diagnostics bundle exec rake test ...
  • This should also let us drop the diagnostic_ prefix from the config vars
  • I find the config/helper suffixes _fields and _config a bit confusing. Perhaps:
    • Call the config hash pipelines_to_run or pipelines_to_test
    • Call the helper method pipeline_to_run so it matches (instead of diagnostic_test_pipeline_config)
    • Move the helper method from DiagnosticsTest class to the PipelineTest class (it's not useful to other diagnostics suites, right?)
In apps/workbench/test/test_helper.rb
  • Perhaps instead of if !ENV["RAILS_ENV"] this should be if ENV["RAILS_ENV"] != "diagnostics" -- we don't want to gain the ability to (presumably accidentally) run tests using development/production configurations, right?

How does rails know not to run the diagnostics tests when I do a regular "bundle exec rake test"? (Is it just that Rails only runs the known types like unit, functional, integration?) Perhaps worth adding a comment (in DiagnosticsTest?) like "`rake test` doesn't run these tests unless requested explicitly with TEST=test/diagnostics/*.rb"?

wait_until_page_has looks like something Capybara is supposed to do for us already. Or is Capybara so clever that it decides "if there is no AJAX request pending and no Javascript running, there is no point waiting any longer for this to appear"?

In apps/workbench/test/diagnostics/pipeline_test.rb
  • select_inut should be select_input

#8 Updated by Radhika Chippada over 6 years ago

Tom,
Updated per you feedback.

- Added "diagnostics" section in application.default.yml with a sample configuration. We no longer need to set it to false in the "common" section.

- It appears that rails does not run the tests in the diagnostics directory when the entire suite is run. I added a comment to DiagnosticsTest.

- Regarding "wait_until_page_has", yes, unless I am looking for something the test will exit after finishing all steps (after starting the pipeline, in these tests). Per our earlier discussion, I hence am waiting for a configured amount of time to see if the pipeline reaches a "Complete" state. Otherwise, we will consider that as a failure.

#9 Updated by Tom Clegg over 6 years ago

LGTM, thanks!

#10 Updated by Radhika Chippada over 6 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:2d04e7d3466cfbf166aecfde71dea909ad0b9119.

Also available in: Atom PDF