Bug #5898: [Tests] Race condition in test_KeepProxyTest1 (tests.test_keep_client.KeepProxyTestCase) e.g. build 1550 - Arvados

Actions

Copy link

Bug #5898

closed

[Tests] Race condition in test_KeepProxyTest1 (tests.test_keep_client.KeepProxyTestCase) e.g. build 1550

Added by Tom Clegg almost 9 years ago. Updated almost 9 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Category:

Tests

Target version:

2015-07-22 sprint

Story points:

0.5

Subtasks 1 (0 open — 1 closed)

Actions

Copy link

Updated by Tom Clegg almost 9 years ago

Suggest a wait loop in run_test_servers.py→_start_keep() and _start_keep_proxy() that waits up to X milliseconds for the port to listen (by reading /proc/net/tcp?) and returns or raises depending on the outcome.

Actions

Copy link

Updated by Brett Smith almost 9 years ago

Target version changed from Bug Triage to Arvados Future Sprints

Actions

Copy link

Updated by Tom Clegg almost 9 years ago

Category set to Tests

These tests are still failing intermittently.

test_KeepAuthenticatedSignedTest
test_KeepAuthenticatedUnsignedTest
test_KeepUnauthenticatedSignedTest
test_KeepProxyTest1
test_KeepProxyTest2

https://ci.curoverse.com/job/arvados-api-server/1772/console

test_KeepAuthenticatedSignedTest (tests.test_keep_client.KeepOptionalPermission) ... ERROR
test_KeepAuthenticatedUnsignedTest (tests.test_keep_client.KeepOptionalPermission) ... ERROR
test_KeepUnauthenticatedSignedTest (tests.test_keep_client.KeepOptionalPermission) ... ERROR
test_KeepUnauthenticatedUnsignedTest (tests.test_keep_client.KeepOptionalPermission) ... 2015/07/07 15:49:41 keepstore starting, pid 20148
2015/07/07 15:49:41 keepstore starting, pid 20147
2015/07/07 15:49:41 Using volume [UnixVolume /tmp/tmpuF7GJk] (writable=true)
2015/07/07 15:49:41 Using volume [UnixVolume /tmp/tmpqHhRGo] (writable=true)
2015/07/07 15:49:41 listening at :39389
2015/07/07 15:49:41 listening at :33496
...
======================================================================
ERROR: test_KeepAuthenticatedSignedTest (tests.test_keep_client.KeepOptionalPermission)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/tests/test_keep_client.py", line 178, in test_KeepAuthenticatedSignedTest
    signed_locator = self._put_foo_and_check()
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/tests/test_keep_client.py", line 170, in _put_foo_and_check
    signed_locator = self.keep_client.put('foo')
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/arvados/retry.py", line 154, in num_retries_setter
    return orig_func(self, *args, **kwargs)
  File "/data/1/jenkins/workspace/arvados-api-server/sdk/python/arvados/keep.py", line 985, in put
    data_hash, copies, thread_limiter.done()), service_errors, label="service")
KeepWriteError: failed to write acbd18db4cc2f85cedef654fccc4a4d8 (wanted 2 copies but wrote 0): service http://localhost:33496/ responded with 0 (7, "couldn't connect to host"); service http://localhost:39389/ responded with 0 (7, "couldn't connect to host")
...

https://ci.curoverse.com/job/arvados-api-server/1776/console is similar.

Actions

Copy link

Updated by Tom Clegg almost 9 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Tom Clegg almost 9 years ago

Target version changed from Arvados Future Sprints to 2015-07-22 sprint

Actions

Copy link

Updated by Nico César almost 9 years ago

review d2b707bb7c6dde27095130296f05e998ab90f3e1

LGTM .. I 'll make timeout=10 since 2 seconds could be too little for a overloaded jenkins to spin up the service. (and preventing failing because of false positives)

Actions

Copy link