Project

General

Profile

Actions

Bug #20032

closed

Flaky test in lib/controller/localdb TestConnectThroughTunnelWithProxyOK

Added by Tom Clegg about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
0.5
Release relationship:
Auto

Description

What next? 10 test lib/controller/localdb -check.vv -check.f=TestConnectThroughTunnelWithProxyOK
...

time="2023-01-31T16:36:33.293372907-05:00" level=info msg="gateway server listening at 127.0.0.1:33459" 
UpdateTunnelURL("https://127.0.0.1:45505")
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == tunnel https://127.0.0.1:45505
connecting to gateway through tunnel
time="2023-01-31T16:36:34.004534318-05:00" level=info msg="tunnel connection 2 started" 
was able to read 0000020c -- success, tunnel is working
PASS: container_gateway_test.go:289: ContainerGatewaySuite.TestConnectThroughTunnelWithProxyOK  4.385s

...
(succeeds a few more times)
...

time="2023-01-31T16:36:40.099281919-05:00" level=info msg="gateway server listening at 127.0.0.1:42371" 
UpdateTunnelURL("https://127.0.0.1:46365")
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
ctr.GatewayAddress == 127.0.0.1:0
connecting to gateway through tunnel
INFO[0008] rpc.Conn.socket: server https://127.0.0.1:46365/arvados/v1/connect/zzzzz-dz642-queuedcontainer/ssh did not switch protocols, got status 503 Service Unavailable
container_gateway_test.go:294:
    s.testConnectThroughTunnel(c, "")
container_gateway_test.go:362:
    c.Assert(err, check.IsNil)
... value *errors.errorString = &errors.errorString{s:"server did not provide a tunnel: 503 Service Unavailable: container gateway is running but tunnel is down"} ("server did not provide a tunnel: 503 Service Unavailable: container gateway is running but t\
unnel is down")

FAIL: container_gateway_test.go:289: ContainerGatewaySuite.TestConnectThroughTunnelWithProxyOK

Subtasks 1 (0 open1 closed)

Task #20043: Review 20032-tunnel-testResolvedBrett Smith02/02/2023Actions
Actions #1

Updated by Tom Clegg about 1 year ago

  • Status changed from New to In Progress

20032-tunnel-test @ e20590d485505f58f7745d74a311ca539c9be940 -- developer-run-tests: #3472

Fixed race between "update container record to working gateway url" and "update container record to non-working gateway url that was stored in that same variable a moment ago". (Confirmed in railsapi test logs that that happened during a failed test run.)

Passed 40x test runs.

Of course, jenkins run failed on a different flaky test, in source:lib/controller.

Actions #2

Updated by Brett Smith about 1 year ago

Tom Clegg wrote in #note-1:

20032-tunnel-test @ e20590d485505f58f7745d74a311ca539c9be940 -- developer-run-tests: #3472

I did a retest at developer-run-tests: #3473 and got a different set of failures, which is bitterly ironic, but means I see no reason not to merge this. Thanks.

Actions #3

Updated by Tom Clegg about 1 year ago

  • Status changed from In Progress to Resolved
Actions #4

Updated by Peter Amstutz about 1 year ago

  • Release set to 57
Actions

Also available in: Atom PDF