singularity build permission denied
Getting this error:
2022-02-14T15:44:13.011645007Z [singularity build /tmp/crunch-run.x1pr8-dz642-nd5m2icieqol3zo.1865379834/keep2196129160/by_uuid/x1pr8-4zz18-zxxvvxctd0khbp1/image.sif docker-archive:///tmp/crunch-run-singularity-396810067/image.tar] 2022-02-14T15:44:13.035750005Z INFO: Starting build... 2022-02-14T15:44:13.035750005Z FATAL: While performing build: conveyor failed to get: error opening file "/tmp/crunch-run-singularity-396810067/image.tar": open /tmp/crunch-run-singularity-396810067/image.tar: permission denied
What seems to be happening is that "singularity build" uses suid to change over to the root user. If crunch-run is launched as a non-root user (and runs arv-mount as non-root), then when singularity changes to root, it can't read from the mount.
The solution is to add "--allow-other" to arv-mount.
+ # singularity needs to be owned by root and suid + chown root /var/lib/arvados/bin/singularity \ + /var/lib/arvados/etc/singularity/singularity.conf \ + /var/lib/arvados/etc/singularity/capability.json \ + /var/lib/arvados/etc/singularity/ecl.toml + chmod u+s /var/lib/arvados/bin/singularityThis is mysterious because
arvados-server installalready installs those files with owner=root -- why do they need to be fixed here?
- This isn't normally how singularity does setuid -- normally .../bin/singularity itself is not setuid, it just invokes .../libexec/singularity/bin/starter-suid (which is setuid) when it needs to elevate privileges -- why does this want to work differently?
IOW, with a normal singularity install, the singularity binary runs as the same user as arv-mount, so allow_other / allow_root isn't needed. But arvbox is doing something different. I expect it would work if we enable allow_root in arvbox, but ideally we would just have arvbox work like a normal singularity install instead.
if [[ "$1" != --no-chown ]] ; then chown arvbox:arvbox -R /usr/local $ARVADOS_CONTAINER_PATH \ /var/lib/passenger /var/lib/postgresql \ /var/lib/nginx /var/log/nginx /etc/ssl/private \ /var/lib/gopath /var/lib/pip /var/lib/npm \ /var/lib/arvados fi mkdir -p /tmp/crunch0 /tmp/crunch1 chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1 # singularity needs to be owned by root and suid chown root /var/lib/arvados/bin/singularity \ /var/lib/arvados/etc/singularity/singularity.conf \ /var/lib/arvados/etc/singularity/capability.json \ /var/lib/arvados/etc/singularity/ecl.toml chmod u+s /var/lib/arvados/bin/singularity
Looks like when arvbox's private dir moved from
chown arvbox:arvbox command added the new dir ($ARVADOS_CONTAINER_PATH is /var/lib/arvados-arvbox) but didn't remove
/var/lib/arvados, so it's accidentally sabotaging
chown arvbox:arvbox -R and then partly/incorrectly repairing it with
chown root command.