[Workbench] diagnostic suite runs against a production instance
#6 Updated by Radhika Chippada over 6 years ago
- Added "apps/workbench/test/diagnostics" test directory
- Added diagnostics testing configuration to application.default.yml with explanation as to how to configure diagnostics testing
- Diagnostics testing currently provides ability to configure pipeline running.
- Added apps/workbench/test/diagnostics/pipeline_test.rb that runs the pipelines configured using the configuration parameter.
- The diagnostics tests can be executed using a command as follows:
- RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb (to run just this test file)
- RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/* (to run all test files in this directory, though currently there is only test file)
- The diagnostics tests will run against the workbench URL configured in the config.
#7 Updated by Tom Clegg over 6 years ago
I like having the diagnostics tests in a separate directory, but I think we should also separate them from the other RAILS_ENV environments:In
- Add a new section called "diagnostics" with the examples enabled (leave them as false in the common section) - this should let us run the diagnostics using
RAILS_ENV=diagnostics bundle exec rake test ...
- This should also let us drop the
diagnostic_prefix from the config vars
- I find the config/helper suffixes
_configa bit confusing. Perhaps:
- Call the config hash
- Call the helper method
pipeline_to_runso it matches (instead of
- Move the helper method from DiagnosticsTest class to the PipelineTest class (it's not useful to other diagnostics suites, right?)
- Call the config hash
- Perhaps instead of
if !ENV["RAILS_ENV"]this should be
if ENV["RAILS_ENV"] != "diagnostics"-- we don't want to gain the ability to (presumably accidentally) run tests using development/production configurations, right?
How does rails know not to run the diagnostics tests when I do a regular "bundle exec rake test"? (Is it just that Rails only runs the known types like unit, functional, integration?) Perhaps worth adding a comment (in DiagnosticsTest?) like "`rake test` doesn't run these tests unless requested explicitly with TEST=test/diagnostics/*.rb"?
- select_inut should be select_input
#8 Updated by Radhika Chippada over 6 years ago
Updated per you feedback.
- Added "diagnostics" section in application.default.yml with a sample configuration. We no longer need to set it to false in the "common" section.
- It appears that rails does not run the tests in the diagnostics directory when the entire suite is run. I added a comment to DiagnosticsTest.
- Regarding "wait_until_page_has", yes, unless I am looking for something the test will exit after finishing all steps (after starting the pipeline, in these tests). Per our earlier discussion, I hence am waiting for a configured amount of time to see if the pipeline reaches a "Complete" state. Otherwise, we will consider that as a failure.