Bug #5898

[Tests] Race condition in test_KeepProxyTest1 (tests.test_keep_client.KeepProxyTestCase) e.g. build 1550

Added by Tom Clegg over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
Tests
Target version:
Start date:
07/14/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Subtasks

Task #6618: Review 5898-listen-raceResolvedTom Clegg

Associated revisions

Revision 8e1477de
Added by Tom Clegg over 6 years ago

Merge branch '5898-listen-race' closes #5898

Revision 12ca8452
Added by Tom Clegg over 6 years ago

Merge branch '5898-listen-race' refs #5898

History

#1 Updated by Tom Clegg over 6 years ago

Suggest a wait loop in run_test_servers.py→_start_keep() and _start_keep_proxy() that waits up to X milliseconds for the port to listen (by reading /proc/net/tcp?) and returns or raises depending on the outcome.

#2 Updated by Brett Smith over 6 years ago

  • Target version changed from Bug Triage to Arvados Future Sprints

#3 Updated by Tom Clegg over 6 years ago

  • Category set to Tests
These tests are still failing intermittently.
  • test_KeepAuthenticatedSignedTest
  • test_KeepAuthenticatedUnsignedTest
  • test_KeepUnauthenticatedSignedTest
  • test_KeepProxyTest1
  • test_KeepProxyTest2

https://ci.curoverse.com/job/arvados-api-server/1772/console

test_KeepAuthenticatedSignedTest (tests.test_keep_client.KeepOptionalPermission) ... ERROR
test_KeepAuthenticatedUnsignedTest (tests.test_keep_client.KeepOptionalPermission) ... ERROR
test_KeepUnauthenticatedSignedTest (tests.test_keep_client.KeepOptionalPermission) ... ERROR
test_KeepUnauthenticatedUnsignedTest (tests.test_keep_client.KeepOptionalPermission) ... 2015/07/07 15:49:41 keepstore starting, pid 20148
2015/07/07 15:49:41 keepstore starting, pid 20147
2015/07/07 15:49:41 Using volume [UnixVolume /tmp/tmpuF7GJk] (writable=true)
2015/07/07 15:49:41 Using volume [UnixVolume /tmp/tmpqHhRGo] (writable=true)
2015/07/07 15:49:41 listening at :39389
2015/07/07 15:49:41 listening at :33496
...
======================================================================
ERROR: test_KeepAuthenticatedSignedTest (tests.test_keep_client.KeepOptionalPermission)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/tests/test_keep_client.py", line 178, in test_KeepAuthenticatedSignedTest
    signed_locator = self._put_foo_and_check()
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/tests/test_keep_client.py", line 170, in _put_foo_and_check
    signed_locator = self.keep_client.put('foo')
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/arvados/retry.py", line 154, in num_retries_setter
    return orig_func(self, *args, **kwargs)
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/arvados/keep.py", line 985, in put
    data_hash, copies, thread_limiter.done()), service_errors, label="service")
KeepWriteError: failed to write acbd18db4cc2f85cedef654fccc4a4d8 (wanted 2 copies but wrote 0): service http://localhost:33496/ responded with 0 (7, "couldn't connect to host"); service http://localhost:39389/ responded with 0 (7, "couldn't connect to host")
...

https://ci.curoverse.com/job/arvados-api-server/1776/console is similar.

#4 Updated by Tom Clegg over 6 years ago

  • Status changed from New to In Progress

#5 Updated by Tom Clegg over 6 years ago

  • Target version changed from Arvados Future Sprints to 2015-07-22 sprint

#6 Updated by Nico César over 6 years ago

review d2b707bb7c6dde27095130296f05e998ab90f3e1

LGTM .. I 'll make timeout=10 since 2 seconds could be too little for a overloaded jenkins to spin up the service. (and preventing failing because of false positives)

#7 Updated by Tom Clegg over 6 years ago

10 seconds, done. Thanks.

#8 Updated by Tom Clegg over 6 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:8e1477dea24ed50b09c055092314fb6522c5a114.

Also available in: Atom PDF