Bug #21055
closed`arvados-server install -type test` fails when trying to edit /etc/sysctl.conf
Description
Initially seen in packer-build-jenkins-image-workbench2-tests: #20 and later confirmed by running make workbench2-build-image
locally. This seems to be related to e8483bb206a from #18874.
... 250.4 Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket. 250.5 Processing triggers for systemd (241-7~deb10u10) ... 250.5 Processing triggers for libc-bin (2.28-10) ... 250.5 + add=fs.inotify.max_user_watches=524288 250.5 + grep -F -- fs.inotify.max_user_watches=524288 /etc/sysctl.conf 250.5 + echo fs.inotify.max_user_watches=524288 250.5 + tee -a /etc/sysctl.conf 250.5 fs.inotify.max_user_watches=524288 250.5 + sysctl -p 250.5 sysctl: setting key "fs.inotify.max_user_watches": Read-only file system 250.5 time="2023-10-10T14:05:25.413604863Z" level=info msg=exiting error="exit status 255" 250.5 exit status 1 ------ Dockerfile:29 -------------------- 28 | # preseed arvados build dependencies 29 | >>> RUN cd /usr/src/arvados && \ 30 | >>> apt-get update && \ 31 | >>> go mod download && \ 32 | >>> go run ./cmd/arvados-server install -type test && cd .. && \ 33 | >>> rm -rf arvados && \ 34 | >>> apt-get clean 35 | -------------------- ERROR: failed to solve: process "/bin/sh -c cd /usr/src/arvados && apt-get update && go mod download && go run ./cmd/arvados-server install -type test && cd .. && rm -rf arvados && apt-get clean" did not complete successfully: exit code: 1 make: *** [workbench2-build-image] Error 1
Found this while trying to rebuild the wb2 test pipeline image.
Updated by Lucas Di Pentima over 1 year ago
It seems that the offending change isn't really necessary, as docker containers inherit the configuration from the host. The jenkins/packer-images/jenkins-image-workbench2.sh
at the arvados-dev
repo already sets this up: https://github.com/arvados/arvados-dev/commit/1d50405ccefb82ed270c858c2033f694ed8d0720
Updated by Lucas Di Pentima over 1 year ago
Updates at 9850ac96ab - branch 21055-arvados-server-install-fix
Test run: developer-run-tests: #3855
Applies the following change so that it isn't a fatal error to not be able to apply sysctl
config changes:
diff --git a/lib/install/deps.go b/lib/install/deps.go
index 263f018097..076a4e2b6a 100644
--- a/lib/install/deps.go
+++ b/lib/install/deps.go
@@ -268,7 +268,9 @@ if ! grep -F -- "$add" /etc/sysctl.conf; then
fi
`, stdout, stderr)
if err != nil {
- return 1
+ // Just warn instead of fail because this is expected when running
+ // inside Docker.
+ logger.Warnf("couldn't set fs.inotify.max_user_watches value: %w", err)
}
}
Updated by Lucas Di Pentima over 1 year ago
Made the following change on WB2's Dockerfile
:
diff --git a/docker/Dockerfile b/docker/Dockerfile
index f529b796..400085d3 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -15,6 +15,7 @@ RUN apt-get update && \
RUN /usr/bin/gem install --no-document fpm
RUN cd /usr/src && git clone git://git.arvados.org/arvados.git && \
cd arvados && \
+ git checkout 21055-arvados-server-install-fix && \
GO_VERSION=$(grep 'goversion =' lib/install/deps.go |awk -F'"' '{print $2}') && \
echo $GO_VERSION && \
cd /usr/src && \
...and was able to run make workbench2-build-image
successfully:
$ make workbench2-build-image (cd docker && docker build -t workbench2-build .) [+] Building 740.2s (11/11) FINISHED docker:desktop-linux => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.63kB 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/node:12.22.3-buster 1.1s => [1/7] FROM docker.io/library/node:12.22.3-buster@sha256:7fbcfcad927d545d7758df291c74a91033bb0a57cdb3c6e 0.0s => CACHED [2/7] RUN echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list.d 0.0s => CACHED [3/7] RUN apt-get update && apt-get -yq --no-install-recommends -o Acquire::Retries=6 instal 0.0s => CACHED [4/7] RUN /usr/bin/gem install --no-document fpm 0.0s => [5/7] RUN cd /usr/src && git clone git://git.arvados.org/arvados.git && cd arvados && git chec 25.4s => [6/7] RUN cd /usr/src/arvados && apt-get update && go mod download && go run ./cmd/arvado 661.7s => [7/7] RUN git config --global --add safe.directory /usr/src/arvados && git config --global --add sa 0.4s => exporting to image 51.6s => => exporting layers 51.6s => => writing image sha256:4f696929c60fbe5659884a06a1389ffbe5ced095a099a7c604974b0b45f3b4fd 0.0s => => naming to docker.io/library/workbench2-build
Updated by Lucas Di Pentima over 1 year ago
Updates at 1c44784440
Failed test rerun: developer-run-tests-remainder: #4051
- Replaces
%w
error wrapping verb with%v
on error logging because of(*github.com/sirupsen/logrus.Logger).Warnf
lack of support for it.
Updated by Peter Amstutz over 1 year ago
- Target version changed from Development 2023-10-11 sprint to Development 2023-10-25 sprint
Updated by Lucas Di Pentima about 1 year ago
- % Done changed from 0 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|305ca4a6af3a93c0c85adadc330ac6b0c6abddf4.
Updated by Tom Clegg about 1 year ago
- Assigned To changed from Lucas Di Pentima to Tom Clegg
- Status changed from Resolved to In Progress
- succeed when running in a docker container and the limit is already high enough, and
- fail when running in a docker container and the limit is too low and we can't fix it.
(If we just ignore the error, then the warning will scroll by and almost certainly be forgotten by the time someone tries to run the workbench2 test suite and gets ENOSPC errors.)
Also fixed a couple of unrelated issues encountered while testing. Running on debian 12 skipped a lot of steps and exited 0, instead of complaining about not knowing how to install docker; now it knows how to install docker on debian 12, and unrecognized debian versions will error out instead of skipping steps.
21055-sysctl-in-docker @ 621cd0a2fc6dd8c1b63eeb687b4c4b7efb130f14
Updated by Tom Clegg about 1 year ago
- % Done changed from 50 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|220869d94387efb179d1692ada16f39049f56700.