Arvados

• Added "apps/workbench/test/diagnostics" test directory

• Added diagnostics testing configuration to application.default.yml with explanation as to how to configure diagnostics testing

• Diagnostics testing currently provides ability to configure pipeline running.

• Added apps/workbench/test/diagnostics/pipeline_test.rb that runs the pipelines configured using the configuration parameter.

• The diagnostics tests can be executed using a command as follows:
  • RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/pipeline_test.rb (to run just this test file)
  • RAILS_ENV=diagnostics bundle exec rake TEST=test/diagnostics/* (to run all test files in this directory, though currently there is only test file)

• The diagnostics tests will run against the workbench URL configured in the config.

I like having the diagnostics tests in a separate directory, but I think we should also separate them from the other RAILS_ENV environments:

• Add a new section called "diagnostics" with the examples enabled (leave them as false in the common section) - this should let us run the diagnostics using RAILS_ENV=diagnostics bundle exec rake test ...
• This should also let us drop the diagnostic_ prefix from the config vars
• I find the config/helper suffixes _fields and _config a bit confusing. Perhaps:
  • Call the config hash pipelines_to_run or pipelines_to_test
  • Call the helper method pipeline_to_run so it matches (instead of diagnostic_test_pipeline_config)
  • Move the helper method from DiagnosticsTest class to the PipelineTest class (it's not useful to other diagnostics suites, right?)

• Perhaps instead of if !ENV["RAILS_ENV"] this should be if ENV["RAILS_ENV"] != "diagnostics" -- we don't want to gain the ability to (presumably accidentally) run tests using development/production configurations, right?

How does rails know not to run the diagnostics tests when I do a regular "bundle exec rake test"? (Is it just that Rails only runs the known types like unit, functional, integration?) Perhaps worth adding a comment (in DiagnosticsTest?) like "`rake test` doesn't run these tests unless requested explicitly with TEST=test/diagnostics/*.rb"?

wait_until_page_has looks like something Capybara is supposed to do for us already. Or is Capybara so clever that it decides "if there is no AJAX request pending and no Javascript running, there is no point waiting any longer for this to appear"?

• select_inut should be select_input

Tom,
Updated per you feedback.

- Added "diagnostics" section in application.default.yml with a sample configuration. We no longer need to set it to false in the "common" section.

- It appears that rails does not run the tests in the diagnostics directory when the entire suite is run. I added a comment to DiagnosticsTest.

- Regarding "wait_until_page_has", yes, unless I am looking for something the test will exit after finishing all steps (after starting the pipeline, in these tests). Per our earlier discussion, I hence am waiting for a configured amount of time to see if the pipeline reaches a "Complete" state. Otherwise, we will consider that as a failure.

LGTM, thanks!