Project

General

Profile

Actions

Bug #21055

closed

`arvados-server install -type test` fails when trying to edit /etc/sysctl.conf

Added by Lucas Di Pentima 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CI
Story points:
-
Release relationship:
Auto

Description

Initially seen in packer-build-jenkins-image-workbench2-tests: #20 and later confirmed by running make workbench2-build-image locally. This seems to be related to e8483bb206a from #18874.

...
250.4 Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
250.5 Processing triggers for systemd (241-7~deb10u10) ...
250.5 Processing triggers for libc-bin (2.28-10) ...
250.5 + add=fs.inotify.max_user_watches=524288
250.5 + grep -F -- fs.inotify.max_user_watches=524288 /etc/sysctl.conf
250.5 + echo fs.inotify.max_user_watches=524288
250.5 + tee -a /etc/sysctl.conf
250.5 fs.inotify.max_user_watches=524288
250.5 + sysctl -p
250.5 sysctl: setting key "fs.inotify.max_user_watches": Read-only file system
250.5 time="2023-10-10T14:05:25.413604863Z" level=info msg=exiting error="exit status 255" 
250.5 exit status 1
------
Dockerfile:29
--------------------
  28 |     # preseed arvados build dependencies
  29 | >>> RUN cd /usr/src/arvados && \
  30 | >>>     apt-get update && \
  31 | >>>     go mod download && \
  32 | >>>     go run ./cmd/arvados-server install -type test && cd .. && \
  33 | >>>     rm -rf arvados && \
  34 | >>>     apt-get clean
  35 |
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /usr/src/arvados &&     apt-get update &&     go mod download &&     go run ./cmd/arvados-server install -type test && cd .. &&     rm -rf arvados &&     apt-get clean" did not complete successfully: exit code: 1
make: *** [workbench2-build-image] Error 1

Found this while trying to rebuild the wb2 test pipeline image.


Subtasks 2 (0 open2 closed)

Task #21056: Review 21055-arvados-server-install-fixResolvedPeter Amstutz10/10/2023Actions
Task #21088: Review 21055-sysctl-in-dockerResolvedTom Clegg10/10/2023Actions
Actions #1

Updated by Lucas Di Pentima 5 months ago

  • Description updated (diff)
Actions #2

Updated by Lucas Di Pentima 5 months ago

  • Description updated (diff)
Actions #4

Updated by Lucas Di Pentima 5 months ago

It seems that the offending change isn't really necessary, as docker containers inherit the configuration from the host. The jenkins/packer-images/jenkins-image-workbench2.sh at the arvados-dev repo already sets this up: https://github.com/arvados/arvados-dev/commit/1d50405ccefb82ed270c858c2033f694ed8d0720

Actions #5

Updated by Lucas Di Pentima 5 months ago

Updates at 9850ac96ab - branch 21055-arvados-server-install-fix
Test run: developer-run-tests: #3855

Applies the following change so that it isn't a fatal error to not be able to apply sysctl config changes:

diff --git a/lib/install/deps.go b/lib/install/deps.go
index 263f018097..076a4e2b6a 100644
--- a/lib/install/deps.go
+++ b/lib/install/deps.go
@@ -268,7 +268,9 @@ if ! grep -F -- "$add" /etc/sysctl.conf; then
 fi
 `, stdout, stderr)
                if err != nil {
-                       return 1
+                       // Just warn instead of fail because this is expected when running
+                       // inside Docker.
+                       logger.Warnf("couldn't set fs.inotify.max_user_watches value: %w", err)
                }
        }
Actions #6

Updated by Lucas Di Pentima 5 months ago

Made the following change on WB2's Dockerfile:

diff --git a/docker/Dockerfile b/docker/Dockerfile
index f529b796..400085d3 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -15,6 +15,7 @@ RUN apt-get update && \
 RUN /usr/bin/gem install --no-document fpm
 RUN cd /usr/src && git clone git://git.arvados.org/arvados.git && \
     cd arvados && \
+    git checkout 21055-arvados-server-install-fix && \
     GO_VERSION=$(grep 'goversion =' lib/install/deps.go |awk -F'"' '{print $2}') && \
     echo $GO_VERSION && \
     cd /usr/src && \

...and was able to run make workbench2-build-image successfully:

$ make workbench2-build-image
(cd docker && docker build -t workbench2-build .)
[+] Building 740.2s (11/11) FINISHED                                                         docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                         0.0s
 => => transferring dockerfile: 1.63kB                                                                       0.0s
 => [internal] load .dockerignore                                                                            0.0s
 => => transferring context: 2B                                                                              0.0s
 => [internal] load metadata for docker.io/library/node:12.22.3-buster                                       1.1s
 => [1/7] FROM docker.io/library/node:12.22.3-buster@sha256:7fbcfcad927d545d7758df291c74a91033bb0a57cdb3c6e  0.0s
 => CACHED [2/7] RUN echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list.d  0.0s
 => CACHED [3/7] RUN apt-get update &&     apt-get -yq --no-install-recommends -o Acquire::Retries=6 instal  0.0s
 => CACHED [4/7] RUN /usr/bin/gem install --no-document fpm                                                  0.0s
 => [5/7] RUN cd /usr/src && git clone git://git.arvados.org/arvados.git &&     cd arvados &&     git chec  25.4s
 => [6/7] RUN cd /usr/src/arvados &&     apt-get update &&     go mod download &&     go run ./cmd/arvado  661.7s
 => [7/7] RUN git config --global --add safe.directory /usr/src/arvados &&     git config --global --add sa  0.4s
 => exporting to image                                                                                      51.6s
 => => exporting layers                                                                                     51.6s
 => => writing image sha256:4f696929c60fbe5659884a06a1389ffbe5ced095a099a7c604974b0b45f3b4fd                 0.0s
 => => naming to docker.io/library/workbench2-build
Actions #7

Updated by Lucas Di Pentima 5 months ago

Updates at 1c44784440
Failed test rerun: developer-run-tests-remainder: #4051

  • Replaces %w error wrapping verb with %v on error logging because of (*github.com/sirupsen/logrus.Logger).Warnf lack of support for it.
Actions #8

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2023-10-11 sprint to Development 2023-10-25 sprint
Actions #9

Updated by Peter Amstutz 5 months ago

LGTM

Actions #10

Updated by Lucas Di Pentima 5 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved
Actions #11

Updated by Tom Clegg 5 months ago

  • Assigned To changed from Lucas Di Pentima to Tom Clegg
  • Status changed from Resolved to In Progress
I have a different fix for this, because I think we want to
  • succeed when running in a docker container and the limit is already high enough, and
  • fail when running in a docker container and the limit is too low and we can't fix it.

(If we just ignore the error, then the warning will scroll by and almost certainly be forgotten by the time someone tries to run the workbench2 test suite and gets ENOSPC errors.)

Also fixed a couple of unrelated issues encountered while testing. Running on debian 12 skipped a lot of steps and exited 0, instead of complaining about not knowing how to install docker; now it knows how to install docker on debian 12, and unrecognized debian versions will error out instead of skipping steps.

21055-sysctl-in-docker @ 621cd0a2fc6dd8c1b63eeb687b4c4b7efb130f14

Actions #12

Updated by Lucas Di Pentima 5 months ago

This LGTM, thanks.

Actions #13

Updated by Peter Amstutz 5 months ago

  • Release set to 67
Actions #15

Updated by Tom Clegg 5 months ago

  • % Done changed from 50 to 100
  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF