Project

General

Profile

Bug #11071

Updated by Tom Morris about 7 years ago

Oldest 'OK' run: https://ci.curoverse.com/job/qr1hi-automated-performance-suite/3162/ 

 Since then (ie Dec. 26, 2016) they have been broken. For example, from https://ci.curoverse.com/job/qr1hi-automated-performance-suite/3427/console: 

 <pre> 
                        ********** Starting performance test ********** 

 Previous HEAD position was aaa45b0... Merge branch '10968-arvput-resume-progress' Closes #10968 
 HEAD is now at 85f8ab2... Bump up rails package iteration. 
 Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`. 
 Using rake 10.4.2 
 Using RedCloth 4.2.9 
 Using i18n 0.7.0 
 Using json 1.8.3 
 Using minitest 5.8.4 
 Using thread_safe 0.3.5 
 Using builder 3.2.2 
 Using erubis 2.7.0 
 Using rack 1.5.5 
 Using mime-types 2.99 
 Using arel 5.0.1.20140414130214 
 Using addressable 2.4.0 
 Using andand 1.3.3 
 Using angularjs-rails 1.3.8 
 Using extlib 0.9.16 
 Using multi_json 1.12.1 
 Using multipart-post 2.0.0 
 Using jwt 1.5.4 
 Using little-plugger 1.1.4 
 Using memoist 0.14.0 
 Using os 0.9.6 
 Using retriable 1.4.1 
 Using sass 3.4.9 
 Using thor 0.19.1 
 Using columnize 0.9.0 
 Using debugger-linecache 1.2.0 
 Using slop 3.6.0 
 Using highline 1.6.21 
 Using net-ssh 2.9.2 
 Using mini_portile 0.6.2 
 Using ffi 1.9.10 
 Using cliver 0.3.2 
 Using coffee-script-source 1.8.0 
 Using execjs 2.7.0 
 Using commonjs 0.2.7 
 Using daemon_controller 1.2.0 
 Using deep_merge 1.0.1 
 Using docile 1.1.5 
 Using headless 1.0.2 
 Using httpclient 2.8.2.4 
 Using libv8 3.16.14.7 
 Using logstash-event 1.2.02 
 Using metaclass 0.0.4 
 Using oj 2.11.2 
 Using bundler 1.11.2 
 Using websocket-extensions 0.1.1 
 Using rails-perftest 0.0.5 
 Using raphael-rails 2.1.2 
 Using ref 1.0.5 
 Using ruby-debug-passenger 0.2.0 
 Using ruby-prof 0.15.2 
 Using rubyzip 1.1.7 
 Using safe_yaml 1.0.4 
 Using tilt 1.4.1 
 Using websocket 1.2.2 
 Using simplecov-html 0.8.0 
 Using sshkey 1.6.1 
 Using wiselinks 1.2.1 
 Using rake-compiler 0.9.5 
 Using tzinfo 1.2.2 
 Using rack-test 0.6.3 
 Using sprockets 3.2.0 
 Using rack-mini-profiler 0.9.2 
 Using mail 2.6.3 
 Using launchy 2.4.3 
 Using autoparse 0.3.3 
 Using faraday 0.9.2 
 Using logging 2.1.0 
 Using bootstrap-sass 3.1.1.1 
 Using byebug 3.5.1 
 Using net-scp 1.2.1 
 Using net-sftp 2.1.2 
 Using net-ssh-gateway 1.2.0 
 Using nokogiri 1.6.6.4 
 Using childprocess 0.5.6 
 Using coffee-script 2.3.0 
 Using uglifier 2.7.2 
 Using less 2.6.0 
 Using passenger 4.0.57 
 Using mocha 1.1.0 
 Using websocket-driver 0.5.1 
 Using therubyracer 0.12.1 
 Using simplecov 0.9.1 
 Using fast_stack 0.1.0 
 Using activesupport 4.1.12 
 Using signet 0.7.2 
 Using capistrano 2.15.5 
 Using xpath 2.0.0 
 Using selenium-webdriver 2.53.1 
 Using simplecov-rcov 0.2.3 
 Using flamegraph 0.1.0 
 Using actionview 4.1.12 
 Using activemodel 4.1.12 
 Using googleauth 0.5.1 
 Using rvm-capistrano 1.5.5 
 Using capybara 2.5.0 
 Using actionpack 4.1.12 
 Using activerecord 4.1.12 
 Using google-api-client 0.8.6 
 Using poltergeist 1.5.1 
 Using actionmailer 4.1.12 
 Using railties 4.1.12 
 Using less-rails 2.6.0 
 Using sprockets-rails 2.3.2 
 Using activerecord-nulldb-adapter 0.3.1 
 Using arvados 0.1.20160420143004 
 Using bootstrap-tab-history-rails 0.1.0 
 Using bootstrap-x-editable-rails 1.5.1.1 
 Using coffee-rails 4.1.0 
 Using jquery-rails 3.1.2 
 Using lograge 0.3.6 
 Using morrisjs-rails 0.5.1 
 Using rails 4.1.12 
 Using sass-rails 5.0.1 
 Using piwik_analytics 1.0.2 
 Using themes_for_rails 0.5.1 from https://github.com/curoverse/themes_for_rails (at master@6115487) 
 Bundle complete! 46 Gemfile dependencies, 116 gems now installed. 
 Use `bundle show [gemname]` to see where a bundled gem is installed. 
 Called 'load' without the :safe option -- defaulting to safe mode. 
 You can avoid this warning in the future by setting the SafeYAML::OPTIONS[:default_mode] option (to :safe or :unsafe). 
 Called 'load' without the :safe option -- defaulting to safe mode. 
 You can avoid this warning in the future by setting the SafeYAML::OPTIONS[:default_mode] option (to :safe or :unsafe). 
 Using display :182 for headless tests 
 Using port 32800 for poltergeist 
 Run options: --seed 41450 

 # Running: 

 BrowsingTest#test_home_page (5.93 sec warmup) 
            wall_time: 5.06 sec 
 .Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g 
 Saved ./tmp/workbench-fail-1.png 
 BrowsingTest#test_search_for_hash (0 ms warmup) 
 Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g 
 Saved ./tmp/workbench-fail-2.png 
 Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g 
 Saved ./tmp/workbench-fail-3.png 
 Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g 
 Saved ./tmp/workbench-fail-4.png 
 Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g 
 Saved ./tmp/workbench-fail-5.png 
 Error during WebSocket handshake: location mismatch: wss://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g != ws://ws.qr1hi.arvadosapi.com/websocket?api_token=5rrtrgnlb2n20qtp2mft5cqpi7psqr6l29mbnmsmtwzra2jt0g 
 Saved ./tmp/workbench-fail-6.png 
            wall_time: 0 ms 
 E 

 Finished in 445.442299s, 0.0045 runs/s, 0.0000 assertions/s. 

   1) Error: 
 BrowsingTest#test_search_for_hash: 
 Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches 
     test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>' 
     test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>' 
     test/test_helper.rb:299:in `run' 

 Error: 
 BrowsingTest#test_search_for_hash: 
 Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches 
     test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>' 
     test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>' 
     test/test_helper.rb:299:in `run' 

 Error: 
 BrowsingTest#test_search_for_hash: 
 Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches 
     test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>' 
     test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>' 
     test/test_helper.rb:299:in `run' 

 Error: 
 BrowsingTest#test_search_for_hash: 
 Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches 
     test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>' 
     test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>' 
     test/test_helper.rb:299:in `run' 

 Error: 
 BrowsingTest#test_search_for_hash: 
 Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches 
     test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>' 
     test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>' 
     test/test_helper.rb:299:in `run' 

 Error: 
 BrowsingTest#test_search_for_hash: 
 Capybara::ExpectationNotMet: expected to find css ".selectable[data-object-uuid]" but there were no matches 
     test/performance/browsing_test.rb:43:in `block (2 levels) in <class:BrowsingTest>' 
     test/performance/browsing_test.rb:40:in `block in <class:BrowsingTest>' 
     test/test_helper.rb:299:in `run' 

 2 runs, 0 assertions, 0 failures, 1 errors, 0 skips 
 Coverage report generated for Unit Tests to /data/1/jenkins/workspace/qr1hi-automated-performance-suite/apps/workbench/coverage. 198 / 473 LOC (41.86%) covered. 
 Coverage report Rcov style generated for Unit Tests to /data/1/jenkins/workspace/qr1hi-automated-performance-suite/apps/workbench/coverage/rcov 
 rake aborted! 
 Command failed with status (1): [ruby -I"lib:test" -I"/usr/local/rvm/gems/ruby-2.1.5/gems/rake-10.4.2/lib" "/usr/local/rvm/gems/ruby-2.1.5/gems/rake-10.4.2/lib/rake/rake_test_loader.rb" "test/performance/**/*_test.rb" ] 
 /usr/local/rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `eval' 
 /usr/local/rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `<main>' 
 Tasks: TOP => test:benchmark 
 (See full trace by running task with --trace) 
 Previous HEAD position was 85f8ab2... Bump up rails package iteration. 
 Switched to branch 'master' 
 Your branch is behind 'origin/master' by 4870 commits, and can be fast-forwarded. 
   (use "git pull" to update your local branch) 
 </pre> 

 The error about websockets is a bit mysterious but also probably a red herring. It may be related to the older version of phantomjs (1.9.8) we are pinned to, at least according to this page: http://www.howtobuildsoftware.com/index.php/how-do/7o0/amazon-web-services-meteor-websocket-phantomjs-error-during-websocket-handshake-location-mismatch-wss-ws- 

 I verified in the logs on qr1hi that the wss:// service is the one being contacted during the test (so it's not trying to connect to the non-tls version). 

 The first diagnostics test that fails (BrowsingTest#test_search_for_hash) could be a legitimate failure: doing the same thing (searching for the word 'hash' in the search box) takes 1.3 minutes according to my browser's debug console. 

 Suggested actions: 

 a) upgrade workbench's bundle to see if that helps with the wss vs ws issue 
 b) fix search performance on qr1hi (for reference, the same search takes just 33s on su92l)

Back