Story #4156

[Tests] Enable cli tests in run-tests.sh, skip any CLI tests that don't pass

Added by Tom Clegg about 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
Tests
Target version:
Start date:
11/18/2014
Due date:
% Done:

100%

Estimated time:
(Total: 0.50 h)
Story points:
1.0

Description

If we enable this in arvados-dev/jenkins/run-tests.sh, there will be no API server or Keep servers running (until #4534).

For the time being, skip any tests that need API/Keep in order to pass.


Subtasks

Task #4581: [Tests] Make cli tests pass or skip themResolvedRadhika Chippada

Task #4582: Review branches: arvados 4156-cli-tests and arvados-dev 4156-enable-cli-testsResolvedTom Clegg

Task #4597: Turn on cli testing in run-tests.shResolvedRadhika Chippada

Task #4702: Review 4156-no-gemfile-lock-for-gemResolved


Related issues

Related to Arvados - Bug #4028: [Crunch] Task stdout appearing in job logs should be labeled "stdout" instead of "stderr"Closed03/07/2017

Associated revisions

Revision ddaad003
Added by Radhika Chippada about 6 years ago

refs #4156
Merge branch '4156-cli-tests'

Revision 4a96daee
Added by Radhika Chippada about 6 years ago

closes #4156
Merge branch '4156-enable-cli-tests'

Revision 4a96daee
Added by Radhika Chippada about 6 years ago

closes #4156
Merge branch '4156-enable-cli-tests'

Revision 5794ce4f
Added by Tom Clegg almost 6 years ago

Merge branch '4156-no-gemfile-lock-for-gem' refs #4156

Revision fce78b35
Added by Tom Clegg almost 6 years ago

Merge branch '4156-bundle-install-user' refs #4156

Revision fce78b35
Added by Tom Clegg almost 6 years ago

Merge branch '4156-bundle-install-user' refs #4156

Revision 4d154281 (diff)
Added by Ward Vandewege almost 6 years ago

Skip two more CLI tests that need a running API server.

refs #4156

History

#1 Updated by Tom Clegg about 6 years ago

  • Category set to Tests
  • Assigned To set to Peter Amstutz

#2 Updated by Tom Clegg about 6 years ago

  • Subject changed from [Tests] Test crunch-job (in local mode) in CLI test suite, enable cli tests in run-tests.sh, skip any other CLI tests that don't pass to [Tests] Enable cli tests in run-tests.sh, skip any CLI tests that don't pass
  • Story points changed from 1.0 to 0.5

#3 Updated by Ward Vandewege about 6 years ago

  • Target version changed from 2014-10-29 sprint to 2014-11-19 sprint

#4 Updated by Tom Clegg about 6 years ago

  • Target version changed from 2014-11-19 sprint to Arvados Future Sprints

#5 Updated by Tom Clegg about 6 years ago

  • Target version changed from Arvados Future Sprints to 2014-11-19 sprint

#6 Updated by Tom Clegg about 6 years ago

  • Assigned To changed from Peter Amstutz to Radhika Chippada

#7 Updated by Ward Vandewege about 6 years ago

  • Story points changed from 0.5 to 1.0

#8 Updated by Radhika Chippada about 6 years ago

Regarding branch 4156-cli-tests:

  • Initially there were 17 failures and I fixed 15 of them. Please let me know if any of these fixes are incorrect.
    • About 10 of them were failing because they were expecting pdh instead of uuid. Updated these tests to look for a collection uuid and are now passing.
    • A couple of them were failing as they were expecting /^ERROR/ and updating them to /Error/ made them pass
    • A couple of them (test_file_to_file_no_overwrite_file and test_file_to_file_no_overwrite_file_in_dir in test_arv-get) were expecting "already exists" error. However, no error is being returned. Commented out the error expectation and the test is now passing.
  • There are 2 tests that are still failing in test_arv-tag. I never used this before and hence could not understand how to address these. Please give me a hint as to what needs to be done. It appears that the first one "test_single_tag_single_obj" should not use --short, but I could not get it to work even after removing this.
    • Based on our conversations, I gather that arv-tag is not yet working or something along those lines. Skipping these two tests for now.

#9 Updated by Radhika Chippada about 6 years ago

I tested my work as follows:

  • Set ARVADOS_API_HOST and ARVADOS_API_TOKEN to point to my dev api server
  • Run my api server
  • Use the following commands to test:

cd ~/arvados/sdk/python/tests
python run_test_server.py start
python run_test_server.py start_keep

cd ~/arvados/sdk/cli
bundle exec rake test

#10 Updated by Radhika Chippada about 6 years ago

  • Status changed from New to In Progress

#11 Updated by Ward Vandewege about 6 years ago

  • Target version changed from 2014-11-19 sprint to 2014-12-10 sprint

#12 Updated by Tom Clegg about 6 years ago

  • Description updated (diff)

#13 Updated by Radhika Chippada about 6 years ago

Skipped all tests that require apiserver / keep. There are 32 such tests out of 48 tests. 2 arv-tag tests are also skipped. Thus, only 14 out of the 48 tests are executed currently.

#14 Updated by Tom Clegg about 6 years ago

Should these change to "Error" like test_nonexistent_block instead of being commented out?

@@ -58,7 +64,7 @@ class TestArvGet < Minitest::Test
     out, err = capture_subprocess_io do
       assert_arv_get false, @@foo_manifest_locator + '/foo', 'tmp/foo'
     end
-    assert_match /^ERROR:/, err
+#    assert_match /^ERROR:/, err
     assert_equal '', out
     assert_equal 'baz', IO.read('tmp/foo')
   end
@@ -70,12 +76,14 @@ class TestArvGet < Minitest::Test
     out, err = capture_subprocess_io do
       assert_arv_get false, @@foo_manifest_locator + '/', 'tmp/'
     end
-    assert_match /^ERROR:/, err
+#    assert_match /^ERROR:/, err
     assert_equal '', out
     assert_equal 'baz', IO.read('tmp/foo')
   end

The rest LGTM, thanks

#15 Updated by Radhika Chippada about 6 years ago

I might have seen it incorrectly during my initial testing and thought that this error is not resulting. Now, I see it work correctly. Updated accordingly. Thanks.

#16 Updated by Radhika Chippada about 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100

Applied in changeset arvados-dev|commit:4a96daeef2dd5a209ec685ee3e1d6864987d1f4f.

#17 Updated by Tom Clegg almost 6 years ago

  • Status changed from Resolved to In Progress

#18 Updated by Brett Smith almost 6 years ago

Reviewing arvados-dev branch at commit:12a59fa

I'm afraid this doesn't work for me with RVM. It reports:

Failures (4):
Fail: cli tests (18s)
Fail: apiserver tests (84s)
Fail: sdk/python tests (254s)
Fail: workbench tests (19s)

The API server, Python SDK, and Workbench test failures all look like this:

 --> Skipping compiling of passenger_native_support.so
 --> Downloading precompiled passenger_native_support.so for the current Ruby interpreter...
     (set PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0 to disable)
     Could not download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/4.0.41/rubyext-ruby-2.1.1-x86_64-linux.tar.gz: The requested URL returned error: 404
     Trying next mirror...
     Could not download https://s3.amazonaws.com/phusion-passenger/binaries/passenger/by_release/4.0.41/rubyext-ruby-2.1.1-x86_64-linux.tar.gz: The requested URL returned error: 403
 --> Continuing without passenger_native_support.so.
Unable to autodetect the currently active RVM gem set name. This could happen if you ran this program using 'sudo' instead of 'rvmsudo'. When using RVM, you're always supposed to use 'rvmsudo' instead of 'sudo!'.

Please try rerunning this program using 'rvmsudo'. If that doesn't help, please contact this program's author for support.
Stopping web server... done
/home/brett/repos/arvados/services/api/test/websocket_runner.rb:10:in `block in _system': passenger returned exit code 1 (RuntimeError)

Here's run-tests' diagnostic output:

WORKSPACE=/home/brett/repos/arvados
PATH=/tmp/tmp.3R54wd3YHD/.gem/ruby/2.1.0/bin:/home/brett/.gem/ruby/2.1.0/bin:/home/brett/firefox31:/home/brett/.rvm/gems/ruby-2.1.1/bin:/home/brett/.rvm/gems/ruby-2.1.1@global/bin:/home/brett/.rvm/rubies/ruby-2.1.1/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/games:/home/brett/.local/bin:/usr/local/go/bin:/usr/local/bin:/usr/local/sbin:/usr/local/games:/home/brett/bin:/home/brett/.rvm/bin:/home/brett/.gem/ruby/2.1.0/bin
Will install dependencies to /home/brett/.gem/ruby/2.1.0
Will install arvados gems to /tmp/tmp.3R54wd3YHD/.gem/ruby/2.1.0
Gem search path is GEM_PATH=/tmp/tmp.3R54wd3YHD/.gem/ruby/2.1.0:/home/brett/.gem/ruby/2.1.0:/home/brett/.rvm/rubies/ruby-2.1.1/lib/ruby/gems/2.1.0

Here's the output of my rvm info:

ruby-2.1.1:

  system:  
    uname:       "Linux brinstar 3.12-0.bpo.1-amd64 #1 SMP Debian 3.12.9-1~bpo70+1 (2014-02-07) x86_64 GNU/Linux" 
    system:      "debian/7/x86_64" 
    bash:        "/bin/bash => GNU bash, version 4.2.37(1)-release (x86_64-pc-linux-gnu)" 
    zsh:         "/bin/zsh => zsh 4.3.17 (x86_64-unknown-linux-gnu)" 

  rvm:
    version:      "rvm 1.25.20 (stable) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]" 
    updated:      "8 months 25 days 56 minutes 3 seconds ago" 
    path:         "/home/brett/.rvm" 

  ruby:
    interpreter:  "ruby" 
    version:      "2.1.1p76" 
    date:         "2014-02-24" 
    platform:     "x86_64-linux" 
    patchlevel:   "2014-02-24 revision 45161" 
    full_version: "ruby 2.1.1p76 (2014-02-24 revision 45161) [x86_64-linux]" 

  homes:
    gem:          "/home/brett/.rvm/gems/ruby-2.1.1" 
    ruby:         "/home/brett/.rvm/rubies/ruby-2.1.1" 

  binaries:
    ruby:         "/home/brett/.rvm/rubies/ruby-2.1.1/bin/ruby" 
    irb:          "/home/brett/.rvm/rubies/ruby-2.1.1/bin/irb" 
    gem:          "/home/brett/.rvm/rubies/ruby-2.1.1/bin/gem" 
    rake:         "/home/brett/.rvm/gems/ruby-2.1.1/bin/rake" 

  environment:
    PATH:         "/home/brett/.rvm/gems/ruby-2.1.1/bin:/home/brett/.rvm/gems/ruby-2.1.1@global/bin:/home/brett/.rvm/rubies/ruby-2.1.1/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/games:/home/brett/.local/bin:/usr/local/go/bin:/usr/local/bin:/usr/local/sbin:/usr/local/games:/home/brett/bin:/home/brett/.rvm/bin:/home/brett/.gem/ruby/2.1.0/bin" 
    GEM_HOME:     "/home/brett/.rvm/gems/ruby-2.1.1" 
    GEM_PATH:     "/home/brett/.rvm/gems/ruby-2.1.1:/home/brett/.rvm/gems/ruby-2.1.1@global" 
    MY_RUBY_HOME: "/home/brett/.rvm/rubies/ruby-2.1.1" 
    IRBRC:        "/home/brett/.rvm/rubies/ruby-2.1.1/.irbrc" 
    RUBYOPT:      "" 
    gemset:       "" 

#19 Updated by Tom Clegg almost 6 years ago

  • Status changed from In Progress to Resolved

#20 Updated by Tom Clegg almost 6 years ago

ruby-2.1.5:

  system:
    uname:       "Linux tom.shell.9tee4.arvadosapi.com 3.14-0.bpo.2-amd64 #1 SMP Debian 3.14.15-2~bpo70+1 (2014-08-21) x86_64 GNU/Linux" 
    system:      "debian/7/x86_64" 
    bash:        "/bin/bash => GNU bash, version 4.2.37(1)-release (x86_64-pc-linux-gnu)" 
    zsh:         " => not installed" 

  rvm:
    version:      "rvm 1.26.4 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]" 
    updated:      "1 hour 54 minutes 34 seconds ago" 
    path:         "/usr/local/rvm" 

  ruby:
    interpreter:  "ruby" 
    version:      "2.1.5p273" 
    date:         "2014-11-13" 
    platform:     "x86_64-linux" 
    patchlevel:   "2014-11-13 revision 48405" 
    full_version: "ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-linux]" 

  homes:
    gem:          "/usr/local/rvm/gems/ruby-2.1.5" 
    ruby:         "/usr/local/rvm/rubies/ruby-2.1.5" 

  binaries:
    ruby:         "/usr/local/rvm/rubies/ruby-2.1.5/bin/ruby" 
    irb:          "/usr/local/rvm/rubies/ruby-2.1.5/bin/irb" 
    gem:          "/usr/local/rvm/rubies/ruby-2.1.5/bin/gem" 
    rake:         "/usr/local/rvm/rubies/ruby-2.1.5/bin/rake" 

  environment:
    PATH:         "/usr/local/rvm/gems/ruby-2.1.5/bin:/usr/local/rvm/gems/ruby-2.1.5@global/bin:/usr/local/rvm/rubies/ruby-2.1.5/bin:/usr/local/rvm/bin:/home/tomclegg/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/go/bin:/usr/local/go/bin" 
    GEM_HOME:     "/usr/local/rvm/gems/ruby-2.1.5" 
    GEM_PATH:     "/usr/local/rvm/gems/ruby-2.1.5:/usr/local/rvm/gems/ruby-2.1.5@global" 
    MY_RUBY_HOME: "/usr/local/rvm/rubies/ruby-2.1.5" 
    IRBRC:        "/usr/local/rvm/rubies/ruby-2.1.5/.irbrc" 
    RUBYOPT:      "" 
    gemset:       "" 

WORKSPACE=/home/tomclegg/src/arvados
PATH=/tmp/tmp.7gulvkJKHj/.gem/ruby/2.1.0/bin:/home/tomclegg/.gem/ruby/2.1.0/bin:/usr/local/rvm/gems/ruby-2.1.5/bin:/usr/local/rvm/gems/ruby-2.1.5@global/bin:/usr/local/rvm/rubies/ruby-2.1.5/bin:/usr/local/rvm/bin:/home/tomclegg/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/go/bin:/usr/local/go/bin
Will install dependencies to /home/tomclegg/.gem/ruby/2.1.0
Will install arvados gems to /tmp/tmp.7gulvkJKHj/.gem/ruby/2.1.0
Gem search path is GEM_PATH=/tmp/tmp.7gulvkJKHj/.gem/ruby/2.1.0:/home/tomclegg/.gem/ruby/2.1.0:/usr/local/rvm/rubies/ruby-2.1.5/lib/ruby/gems/2.1.0
New python executable in /tmp/tmp.WgsCC2pNJn/bin/python
Installing setuptools, pip...done.

Also available in: Atom PDF