Arvados

15370-loopback-dispatchcloud @ 2ca82cf645eb7d9dad60f98e1feca67042c38c47 -- developer-run-tests: #3126

Now tests are failing because the CI image doesn't have docker, so "arv-keepdocker" doesn't work.

Added docker install recipe to arvados-server install

15370-loopback-dispatchcloud @ f07c059fca954e4d001cbf1cb36c845be9d884dd

15370-loopback-dispatchcloud @ d6497b0fc0464447d6b753809807a1d11d511e50 -- developer-run-tests: #3136

15370-loopback-dispatchcloud @ d6497b0fc0464447d6b753809807a1d11d511e50 -- developer-run-tests: #3145

15370-loopback-dispatchcloud @ 731c5e81f5aedc82d03786670610bde68bba27c7 -- developer-run-tests: #3146

15370-install-docker @ 663f3742a80b1b236d727d2d27068d03a37b4469

Tom Clegg wrote:

15370-install-docker @ 663f3742a80b1b236d727d2d27068d03a37b4469

LGTM thanks!

Tom Clegg wrote:

15370-loopback-dispatchcloud @ 731c5e81f5aedc82d03786670610bde68bba27c7 -- developer-run-tests: #3146

I updated the jenkins satellite image to incorporate the changes from main, which means docker should now be present. Running these tests again:

developer-run-tests: #3153

That failed because the jenkins user can't access Docker. I pushed the update that adds docker to the 'test' image, and gives the jenkins user access to Docker, and rebuilt the image once more.

developer-run-tests: #3154

Some different failures here:

developer-run-tests-remainder: #3301 /consoleFull

time="2022-05-20T18:30:33.430743543Z" level=error msg=failed error="Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39" 
14:30:33 exit status 1
14:30:33 
14:30:33 ----------------------------------------------------------------------
14:30:33 FAIL: build_test.go:27: BuildSuite.TestBuildAndInstall
14:30:33 
14:30:33 build_test.go:47:
14:30:33     c.Check(err, check.IsNil)
14:30:33 ... value *exec.ExitError = &exec.ExitError{ProcessState:(*os.ProcessState)(0xc00000e0c0), Stderr:[]uint8(nil)} ("exit status 1")
14:30:33 
14:30:33 build_test.go:50:
14:30:33     c.Assert(err, check.IsNil)
14:30:33 ... value *fs.PathError = &fs.PathError{Op:"stat", Path:"/tmp/check-5577006791947779410/0/arvados-server-easy_1.2.3~rc4_amd64.deb", Err:0x2} ("stat /tmp/check-5577006791947779410/0/arvados-server-easy_1.2.3~rc4_amd64.deb: no such file or directory")
14:30:33 
14:30:33 OOPS: 0 passed, 1 FAILED
14:30:33 --- FAIL: Test (0.72s)

Hmm, we were using an old Buster base image. I've bumped it to the latest and am re-building the image now. Maybe that will fix the version difference? I've also added user_allow_other to /etc/fuse.conf in the image, which should fix the other problem in lib/crunchrun tests:

packer-build-jenkins-image-arvados-tests: #82

Here we go:

developer-run-tests: #3155

I suspect adding a non-functional docker may have broken the main branch build by failing tests that were skipped when there was no docker in PATH. Might need to c.Skip() the arvados-package test for the time being. Checking:

15370-docker-tests @ 36cfafd6e7eae2784c22aefdd9df26783412d42a -- developer-run-tests: #3156

Looks like the same applies to integrationSuite.TestRunTrivialContainerWithLocalKeepstore in lib/crunchrun.

15370-docker-tests @ 6caeb0768adabd32b50cc2ca6eb49d162745c4b0 -- developer-run-tests: #3157

(only wb1 integration tests failed there)

Merged main:

15370-loopback-dispatchcloud @ 3fa6aa4043286ad61e5f29c136d3cc2942e8750d -- developer-run-tests: #3158

Looks like I have some more work to do.

The cmd/arvados-package test (which we used to skip because it requires docker) fails because it takes longer than 10m. I updated run-tests.sh to change the timeout to 20m for that suite, but I also updated the jenkins config to skip it in [developer-]run-tests-remainder. We can re-enable it after either (a) changing the image prep so a new jenkins worker has a cached build image (which makes the cmd/arvados-package test run much faster) or (b) moving it to a separate run-tests-package / developer-run-tests-package jenkins job.

Also fixed a "missing keep data dir" testing bug, a docker client usage bug, and a flaky error log test.

15370-loopback-dispatchcloud @ bad877eb1d1a84d25c1fab3592e4218774816179 -- developer-run-tests: #3162

retry wb1 developer-run-tests-apps-workbench-integration: #3387

Tom Clegg wrote:

The cmd/arvados-package test (which we used to skip because it requires docker) fails because it takes longer than 10m. I updated run-tests.sh to change the timeout to 20m for that suite, but I also updated the jenkins config to skip it in [developer-]run-tests-remainder. We can re-enable it after either (a) changing the image prep so a new jenkins worker has a cached build image (which makes the cmd/arvados-package test run much faster) or (b) moving it to a separate run-tests-package / developer-run-tests-package jenkins job.

Also fixed a "missing keep data dir" testing bug, a docker client usage bug, and a flaky error log test.

15370-loopback-dispatchcloud @ bad877eb1d1a84d25c1fab3592e4218774816179 -- developer-run-tests: #3162

retry wb1 developer-run-tests-apps-workbench-integration: #3387

Is there a reason to pin on a docker API version that is so old? Latest is 1.41, and we're pinning on 1.21.

Otherwise, LGTM, thanks.

Ward Vandewege wrote:

Is there a reason to pin on a docker API version that is so old? Latest is 1.41, and we're pinning on 1.21.

Sort of. I just figured old API versions are supported for a long time, so there's no particular hurry to use a newer one, in which case we might as well use the same version we use in crunch-run.

If someone wants to use docker 1.9 to build packages, who am I to say no...

Tom Clegg wrote:

Ward Vandewege wrote:

Is there a reason to pin on a docker API version that is so old? Latest is 1.41, and we're pinning on 1.21.

Sort of. I just figured old API versions are supported for a long time, so there's no particular hurry to use a newer one, in which case we might as well use the same version we use in crunch-run.

If someone wants to use docker 1.9 to build packages, who am I to say no...

Hmm, actually docker 1.9 would be a problem, the on-disk image format is different (we went through that whole painful migration in #8568 etc). I don't think anyone is using docker that old anymore.

If we're really going to default to a api version that old, there should be a comment in the code that states there is no actual reason for this, only a desire for maximal backwards compatibility.

This would avoid future concern about upping the API version when - inevitably - we'll run into a version of the Docker Engine that doesn't work with an API that old anymore.

• Debian 10 (buster) ships with Docker 1.18.09 which has Docker API 1.39
• Ubuntu 18.04 (bionic) originally shipped with Docker 1.17.12 which has Docker API 1.35

Of course we use the docker package repos to install more recent versions of Docker. Even on CentOS 7 it seems that a recent docker is easily installed, cf. https://docs.docker.com/engine/install/centos/.

15370-loopback-dispatchcloud @ bac1772ab074713e3c50632a4cad3cc1ce50d0ec -- developer-run-tests: #3163

updated crunch-run to docker API 1.35 and exported it as a const so arvados-package can stay in sync.

Tom Clegg wrote:

15370-loopback-dispatchcloud @ bac1772ab074713e3c50632a4cad3cc1ce50d0ec -- developer-run-tests: #3163

updated crunch-run to docker API 1.35 and exported it as a const so arvados-package can stay in sync.

Thank you that's great. LGTM!