Bug #7228
closed
[Crunch] crunch-dispatch should not create tmp files that break API server
Description
Problem¶
We have seen crunch-dispatch break API server as follows:- Run as root, as described in docs
- Call some part of the API server's code base that uses Rails.cache
- Create files and directories in
{Rails.root}/tmp/cachewith owner=root and permissions that prohibit www-data from writing
After this has happened, API server (running as www-data) crashes when trying to update cached data.
This isn't very common because API server usually creates/updates a given cache item before crunch-dispatch does. But when it does happen, it's bad: for example, new groups can't be created because the group cache can't be updated.
The condition can be fixed temporarily by running arvados-api-server-upgrade.sh (it does chown -R on the tmp dir, among other things). However, this doesn't prevent it from happening again.
The real solution is #5162: refactor crunch-dispatch as an API client so it can't touch the API server Rails project at all.
Ideas¶
In the meantime, there might be an effective workaround, like running crunch-dispatch with umask=002 and the same GID as the API server process.
(Running crunch-dispatch with the same UID as the API server process would fix the cache permission issue, but at the cost of introducing other problems: crunch-dispatch needs to use sudo, and giving www-data passwordless sudo undermines the security benefit of running the web service as non-root in the first place.)
Immediate fix¶
arvados-api-server-upgrade.shalready makes $WWW_OWNER the owner oftmp/recursively. Extend it to chmodtmp/cache/2775.- Extend crunch-dispatch to run with a 002 umask. The only other file it opens is its own lockfile, and it sets a specific 0644 mode for that, so this should only affect Rails cache files.
- Test using the procedure in note-5.
- Make sure the arvados-dev branch gets merged before the arvados branch, so we build a new package that includes both the new upgrade script and the new crunch-dispatch.
Updated by 
Updated by 
Updated by