Bug #18732
closed
singularity build permission denied
Added by Peter Amstutz about 2 years ago.
Updated about 2 years ago.
Release relationship:
Auto
Description
Getting this error:
2022-02-14T15:44:13.011645007Z [singularity build /tmp/crunch-run.x1pr8-dz642-nd5m2icieqol3zo.1865379834/keep2196129160/by_uuid/x1pr8-4zz18-zxxvvxctd0khbp1/image.sif docker-archive:///tmp/crunch-run-singularity-396810067/image.tar]
2022-02-14T15:44:13.035750005Z INFO: Starting build...
2022-02-14T15:44:13.035750005Z FATAL: While performing build: conveyor failed to get: error opening file "/tmp/crunch-run-singularity-396810067/image.tar": open /tmp/crunch-run-singularity-396810067/image.tar: permission denied
What seems to be happening is that "singularity build" uses suid to change over to the root user. If crunch-run is launched as a non-root user (and runs arv-mount as non-root), then when singularity changes to root, it can't read from the mount.
The solution is to add "--allow-other" to arv-mount.
- Status changed from New to In Progress
- Assigned To set to Peter Amstutz
- Category set to Crunch
- Description updated (diff)
- Target version changed from 2022-02-16 sprint to 2022-03-02 sprint
- Target version changed from 2022-03-02 sprint to 2022-03-16 sprint
- Target version changed from 2022-03-16 sprint to 2022-03-30 Sprint
- Assigned To changed from Peter Amstutz to Tom Clegg
In #18238, 25bcd259cf59d51263d74ab09dfc4d9a92ca3fa4 adds this:
+ # singularity needs to be owned by root and suid
+ chown root /var/lib/arvados/bin/singularity \
+ /var/lib/arvados/etc/singularity/singularity.conf \
+ /var/lib/arvados/etc/singularity/capability.json \
+ /var/lib/arvados/etc/singularity/ecl.toml
+ chmod u+s /var/lib/arvados/bin/singularity
This is mysterious because
arvados-server install already installs those files with owner=root -- why do they need to be fixed here?- This isn't normally how singularity does setuid -- normally .../bin/singularity itself is not setuid, it just invokes .../libexec/singularity/bin/starter-suid (which is setuid) when it needs to elevate privileges -- why does this want to work differently?
IOW, with a normal singularity install, the singularity binary runs as the same user as arv-mount, so allow_other / allow_root isn't needed. But arvbox is doing something different. I expect it would work if we enable allow_root in arvbox, but ideally we would just have arvbox work like a normal singularity install instead.
if [[ "$1" != --no-chown ]] ; then
chown arvbox:arvbox -R /usr/local $ARVADOS_CONTAINER_PATH \
/var/lib/passenger /var/lib/postgresql \
/var/lib/nginx /var/log/nginx /etc/ssl/private \
/var/lib/gopath /var/lib/pip /var/lib/npm \
/var/lib/arvados
fi
mkdir -p /tmp/crunch0 /tmp/crunch1
chown crunch:crunch -R /tmp/crunch0 /tmp/crunch1
# singularity needs to be owned by root and suid
chown root /var/lib/arvados/bin/singularity \
/var/lib/arvados/etc/singularity/singularity.conf \
/var/lib/arvados/etc/singularity/capability.json \
/var/lib/arvados/etc/singularity/ecl.toml
chmod u+s /var/lib/arvados/bin/singularity
Looks like when arvbox's private dir moved from /var/lib/arvados to /var/lib/arvados-arvbox, that chown arvbox:arvbox command added the new dir ($ARVADOS_CONTAINER_PATH is /var/lib/arvados-arvbox) but didn't remove /var/lib/arvados, so it's accidentally sabotaging /var/lib/arvados with chown arvbox:arvbox -R and then partly/incorrectly repairing it with chown root command.
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by