Bug #21055
closed
`arvados-server install -type test` fails when trying to edit /etc/sysctl.conf
Added by Lucas Di Pentima 7 months ago.
Updated 7 months ago.
Release relationship:
Auto
Description
Initially seen in packer-build-jenkins-image-workbench2-tests: #20 
and later confirmed by running make workbench2-build-image locally. This seems to be related to e8483bb206a from #18874.
...
250.4 Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
250.5 Processing triggers for systemd (241-7~deb10u10) ...
250.5 Processing triggers for libc-bin (2.28-10) ...
250.5 + add=fs.inotify.max_user_watches=524288
250.5 + grep -F -- fs.inotify.max_user_watches=524288 /etc/sysctl.conf
250.5 + echo fs.inotify.max_user_watches=524288
250.5 + tee -a /etc/sysctl.conf
250.5 fs.inotify.max_user_watches=524288
250.5 + sysctl -p
250.5 sysctl: setting key "fs.inotify.max_user_watches": Read-only file system
250.5 time="2023-10-10T14:05:25.413604863Z" level=info msg=exiting error="exit status 255"
250.5 exit status 1
------
Dockerfile:29
--------------------
28 | # preseed arvados build dependencies
29 | >>> RUN cd /usr/src/arvados && \
30 | >>> apt-get update && \
31 | >>> go mod download && \
32 | >>> go run ./cmd/arvados-server install -type test && cd .. && \
33 | >>> rm -rf arvados && \
34 | >>> apt-get clean
35 |
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /usr/src/arvados && apt-get update && go mod download && go run ./cmd/arvados-server install -type test && cd .. && rm -rf arvados && apt-get clean" did not complete successfully: exit code: 1
make: *** [workbench2-build-image] Error 1
Found this while trying to rebuild the wb2 test pipeline image.
- Description updated (diff)
- Description updated (diff)
Updates at 9850ac96ab - branch 21055-arvados-server-install-fix
Test run: developer-run-tests: #3855 
Applies the following change so that it isn't a fatal error to not be able to apply sysctl config changes:
diff --git a/lib/install/deps.go b/lib/install/deps.go
index 263f018097..076a4e2b6a 100644
--- a/lib/install/deps.go
+++ b/lib/install/deps.go
@@ -268,7 +268,9 @@ if ! grep -F -- "$add" /etc/sysctl.conf; then
fi
`, stdout, stderr)
if err != nil {
- return 1
+ // Just warn instead of fail because this is expected when running
+ // inside Docker.
+ logger.Warnf("couldn't set fs.inotify.max_user_watches value: %w", err)
}
}
Made the following change on WB2's Dockerfile:
diff --git a/docker/Dockerfile b/docker/Dockerfile
index f529b796..400085d3 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -15,6 +15,7 @@ RUN apt-get update && \
RUN /usr/bin/gem install --no-document fpm
RUN cd /usr/src && git clone git://git.arvados.org/arvados.git && \
cd arvados && \
+ git checkout 21055-arvados-server-install-fix && \
GO_VERSION=$(grep 'goversion =' lib/install/deps.go |awk -F'"' '{print $2}') && \
echo $GO_VERSION && \
cd /usr/src && \
...and was able to run make workbench2-build-image successfully:
$ make workbench2-build-image
(cd docker && docker build -t workbench2-build .)
[+] Building 740.2s (11/11) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.63kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/node:12.22.3-buster 1.1s
=> [1/7] FROM docker.io/library/node:12.22.3-buster@sha256:7fbcfcad927d545d7758df291c74a91033bb0a57cdb3c6e 0.0s
=> CACHED [2/7] RUN echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list.d 0.0s
=> CACHED [3/7] RUN apt-get update && apt-get -yq --no-install-recommends -o Acquire::Retries=6 instal 0.0s
=> CACHED [4/7] RUN /usr/bin/gem install --no-document fpm 0.0s
=> [5/7] RUN cd /usr/src && git clone git://git.arvados.org/arvados.git && cd arvados && git chec 25.4s
=> [6/7] RUN cd /usr/src/arvados && apt-get update && go mod download && go run ./cmd/arvado 661.7s
=> [7/7] RUN git config --global --add safe.directory /usr/src/arvados && git config --global --add sa 0.4s
=> exporting to image 51.6s
=> => exporting layers 51.6s
=> => writing image sha256:4f696929c60fbe5659884a06a1389ffbe5ced095a099a7c604974b0b45f3b4fd 0.0s
=> => naming to docker.io/library/workbench2-build
- Target version changed from Development 2023-10-11 sprint to Development 2023-10-25 sprint
- % Done changed from 0 to 100
- Status changed from In Progress to Resolved
- Assigned To changed from Lucas Di Pentima to Tom Clegg
- Status changed from Resolved to In Progress
I have a different fix for this, because I think we want to
- succeed when running in a docker container and the limit is already high enough, and
- fail when running in a docker container and the limit is too low and we can't fix it.
(If we just ignore the error, then the warning will scroll by and almost certainly be forgotten by the time someone tries to run the workbench2 test suite and gets ENOSPC errors.)
Also fixed a couple of unrelated issues encountered while testing. Running on debian 12 skipped a lot of steps and exited 0, instead of complaining about not knowing how to install docker; now it knows how to install docker on debian 12, and unrecognized debian versions will error out instead of skipping steps.
21055-sysctl-in-docker @ 621cd0a2fc6dd8c1b63eeb687b4c4b7efb130f14
- % Done changed from 50 to 100
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by